Control device, imaging system, control method, and control program

Abstract

Provided are a control device, an imaging system, a control method, and a control program with which it is possible to capture the image of a mobile object keeping the mobile object always within the view angle of an imaging device.　This control device (60) comprises a CPU6 (0A) that controls each of a camera (10) and a turning mechanism (16) for turning the camera (10). The CPU6 (0A) detects a mobile object from a captured image to which captured image data acquired from the camera (10) is outputted, acquires first position information relating to the position of the mobile object in the captured image, acquires second position information relating to the turning position of the turning mechanism (16), and performs turning control of the turning mechanism (16) on the basis of the first position information and second position information so that the mobile object is included in the imaging range of the camera (10). Detection of the mobile object is executed even during the turning motion of the turning mechanism (16).

Description

Control device, imaging system, control method, and control program

[0001] The present invention relates to a control device, an imaging system, a control method, and a control program.

[0002] Patent Document 1 describes an automatic tracking and photography system that includes a control terminal having a target position prediction unit that predicts the position of the target after a control delay time, and a control amount calculation unit that calculates the control amounts of pan and tilt of the camera required to move the center of the photography screen to the predicted target position, and in which the control terminal remotely controls the camera to automatically track and photograph the target.

[0003] Patent Document 2 describes an automatic tracking device that includes a position storage means that converts the positions of the tracking points of an object to be tracked at different times into positions on absolute coordinates using pan and tilt angles acquired from a rotation means and then stores them in chronological order, a speed calculation means and a position estimation means that estimate the moving speed and the position of the tracking point at the next time based on the positions of the tracking points on the absolute coordinates stored in chronological order, and a rotation control means that calculates pan and tilt control values ​​that can capture the position of the tracking point at the next time in the center of an image and feedback controls the rotation means.

[0004] Patent document 3 describes a remote control pan-and-tilt head system that measures the time it takes for a pan-and-tilt head controller to send a dummy signal to the pan-and-tilt head and receive a response signal from the pan-and-tilt head, and based on this time, sets a delay waiting time for the pan-and-tilt head controller to wait for receiving a response signal from the pan-and-tilt head after sending a control signal to the pan-and-tilt head, thereby setting a highly reliable delay waiting time according to the actual line conditions.

[0005] International Publication No. 2016 / 151925 Japanese Patent Application Publication No. 2001-285850 Japanese Patent Application Publication No. 2001-333320

[0006] One embodiment of the technique of the present disclosure provides a control device, an imaging system, a control method, and a control program that are capable of capturing an image of a moving object so that the moving object does not deviate from the angle of view of the imaging device.

[0007] (1) A control device including a processor that controls an imaging device and a rotation device that rotates the imaging device, wherein the processor: detects a moving object from an image captured by outputting image data acquired from the imaging device; acquires first position information regarding the position of the moving object in the image; acquires second position information regarding the rotation position of the rotation device; and controls the rotation of the rotation device based on the first position information and the second position information so that the moving object is included in the imaging range of the imaging device; and the detection is also performed during the rotation operation of the rotation device.

[0008] (2) The control device according to (1), in which the detection is performed in parallel with the turning control.

[0009] (3) The control device according to (1), wherein the imaging device performs imaging control relating to the imaging range.

[0010] (4) The control device according to (1), in which the detection is performed during a period including the start to completion of the turning control.

[0011] (5) The control device according to (4), wherein the completion of the turning control is a state in which a change in the captured image due to the turning control has ended.

[0012] (6) The control device according to (1), in which the detection is performed during a period between control commands for the turning control.

[0013] (7) The control device according to any one of (1) to (6), in which the detection is repeatedly performed in parallel with the turning control.

[0014] (8) The control device according to (7), in which the detection is performed every time the captured image data is acquired by the imaging device.

[0015] (9) A control device described in any one of (1) to (8), wherein the processor: acquires a third movement vector, which is the movement vector of the moving body in the space in which the moving body exists, based on a first movement vector, which is the movement vector of the moving body in the imaging range, and a second movement vector, which is the movement vector of the imaging range due to the rotation control; acquires third position information regarding the position of the moving body based on the third movement vector; and performs rotation control of the rotation device based on the third position information.

[0016] (10) The control device according to (9), in which the processor acquires the first movement vector and the second movement vector from the captured image.

[0017] (11) The control device according to (10), in which the processor acquires the first movement vector and the second movement vector based on a feature point of the captured image.

[0018] (12) The control device according to (11), in which the processor acquires the second movement vector based on the feature point determined by history information of the turning control.

[0019] (13) The control device according to (9), wherein the processor acquires the second movement vector using a control command for the turning control and history information of the turning control.

[0020] (14) A control device according to any one of (1) to (13), wherein the processor performs a second rotation control of the rotation device at a point before the change in the captured image due to the first rotation control of the rotation device is completed.

[0021] (15) The control device according to (14), wherein the processor performs the second turning control based on a fourth movement vector of the imaging range based on a control value of the first turning control and a fifth movement vector of the imaging range based on the captured image.

[0022] (16) The control device according to any one of (1) to (15), wherein the processor determines a rotation speed of the rotation device according to an imaging condition of the imaging device.

[0023] (17) The control device according to any one of (1) to (16), which is connected to the imaging device and the rotation device via a network.

[0024] (18) An imaging system including an imaging device, a rotation device for rotating the imaging device, and a control device for controlling the imaging device and the rotation device, wherein the control device detects a moving object from a captured image output from image data acquired from the imaging device, acquires first position information relating to the position of the moving object in the captured image, acquires second position information relating to the rotation position of the rotation device, and controls the rotation of the rotation device based on the first position information and the second position information so that the moving object is included in the imaging range of the imaging device, and the detection is also performed during the rotation operation of the rotation device.

[0025] (19) A control method using a control device having a processor that controls an imaging device and a turning device that turns the imaging device, wherein the processor: detects a moving object from an image captured by outputting image data acquired from the imaging device; acquires first position information regarding the position of the moving object in the image captured; acquires second position information regarding the turning position of the turning device; and controls the turning of the turning device based on the first position information and the second position information so that the moving object is included in the imaging range of the imaging device; and the detection is also performed during the turning operation of the turning device.

[0026] (20) A control program for a control device having a processor that controls an imaging device and a rotation device that rotates the imaging device, causing the processor to execute the following processes: detect a moving object from an image captured by outputting image data acquired from the imaging device; acquire first position information regarding the position of the moving object in the image captured; acquire second position information regarding the rotation position of the rotation device; and control the rotation of the rotation device based on the first position information and the second position information so that the moving object is included in the imaging range of the imaging device; and the detection is also performed during the rotation operation of the rotation device.

[0027] According to the present invention, it is possible to provide a control device, an imaging system, a control method, and a control program that are capable of capturing an image of a moving object so that the moving object does not deviate from the angle of view of the imaging device.

[0028] FIG. 1 is a diagram illustrating an example of an imaging system 1 equipped with a control device 60 according to this embodiment. FIG. 2 is a diagram illustrating an example of rotation of the camera 10 in the pitch direction by the rotation mechanism 16. FIG. 3 is a diagram illustrating an example of rotation of the camera 10 in the yaw direction by the rotation mechanism 16. FIG. 4 is a block diagram illustrating an example of the optical system and electrical system configuration of the camera 10. FIG. 5 is a diagram illustrating an example of the electrical system configuration of the rotation mechanism 16 and the management device 11. FIG. 6 is a diagram illustrating, for reference, an example of management processing in a conventional imaging system. FIG. 7 is a diagram illustrating an example of management processing by the imaging system 1 according to this embodiment. FIG. 8 is a diagram illustrating an example of movement position prediction based on the movement vector of a moving object. FIG. 9 is a diagram illustrating an example of movement position prediction based on feature points determined by history information. FIG. 10 is a diagram illustrating a modified example of the management processing by the imaging system 1 according to this embodiment. FIG. 11 is a diagram illustrating an example of how an information processing program for management control is installed in the control device 60 of the management device 11 from a storage medium storing an information processing program for management control.

[0029] Hereinafter, an example of an embodiment of the present invention will be described with reference to the drawings.

[0030] <Imaging System of the Embodiment> Fig. 1 is a diagram showing an example of an imaging system 1 equipped with a control device of the present embodiment. As shown in Fig. 1 as an example, the imaging system 1 includes a camera 10 and a management device 11. The camera 10 and the management device 11 are connected via a network 20. The camera 10 is an example of an imaging device of the present invention.

[0031] The camera 10 is a camera for capturing images of moving objects such as drones, vehicles, and people. A camera capable of telephoto shooting, an ultra-high resolution camera, or the like is used as the camera 10. A wide-angle camera may also be used as the camera 10. The camera 10 is installed via a rotation mechanism 16 (described later) and captures an image of a subject to be captured. The camera 10 transmits the captured image and imaging information related to the image to the management device 11 via the network 20.

[0032] The management device 11 includes a display 13a, a keyboard 13b, a mouse 13c, and a secondary storage device 14. Examples of the display 13a include a liquid crystal display, a plasma display, an organic EL (Electro-Luminescence) display, and a CRT (Cathode Ray Tube) display.

[0033] An example of the secondary storage device 14 is a hard disk drive (HDD). The secondary storage device 14 is not limited to an HDD, and may be any non-volatile memory such as a flash memory, a solid state drive (SSD), or an electrically erasable and programmable read-only memory (EEPROM).

[0034] The management device 11 receives the captured images and imaging information transmitted from the camera 10 , and displays the received captured images and imaging information on the display 13 a and stores them in the secondary storage device 14 .

[0035] The management device 11 performs imaging control to control imaging by the camera 10. For example, the management device 11 performs imaging control by communicating with the camera 10 via the network 20. The imaging control is a control to set imaging parameters for the camera 10 to capture images in the camera 10 and cause the camera 10 to capture images. The imaging parameters include parameters related to exposure and parameters for the zoom position.

[0036] The management device 11 also controls the rotation mechanism 16 to control (pan and tilt) the imaging direction of the camera 10. For example, the management device 11 sets the rotation direction, rotation amount, rotation speed, etc. of the camera 10 in response to operation of the keyboard 13b or mouse 13c, or touch operation on the screen of the display 13a. The management device 11 is connected to the rotation mechanism 16 via the network 20. The management device 11 controls the rotation mechanism 16 via the network 20.

[0037] <Rotation of camera 10 by rotation mechanism 16> Figure 2 is a diagram showing an example of rotation of camera 10 in the pitch direction by rotation mechanism 16. Figure 3 is a diagram showing an example of rotation of camera 10 in the yaw direction by rotation mechanism 16. Camera 10 is attached to rotation mechanism 16. Rotation mechanism 16 can change the imaging direction of camera 10 by rotating camera 10. Rotation mechanism 16 is an example of a rotation device in the present invention.

[0038] Specifically, as shown in Fig. 2 as an example, the rotation mechanism 16 is a two-axis rotation mechanism that can rotate the camera 10 in a rotation direction (pitch direction) that intersects with the yaw direction and has a pitch axis PA as the central axis, and in a rotation direction (yaw direction) that has a yaw axis YA as the central axis, as shown in Fig. 3 as an example. Note that, although the rotation mechanism 16 according to the present embodiment is an example of a two-axis rotation mechanism, the technology of the present disclosure is not limited to this, and may be a three-axis rotation mechanism or a single-axis rotation mechanism.

[0039] <Configuration of Optical System and Electrical System of Camera 10> FIG. 4 is a block diagram showing an example of the configuration of the optical system and electrical system of the camera 10. As shown in FIG. 4, the camera 10 includes an optical system 15 and an image sensor 25. The image sensor 25 is located downstream of the optical system 15. The optical system 15 includes an objective lens 15A and a lens group 15B. The objective lens 15A and the lens group 15B are arranged in this order along the optical axis OA of the optical system 15 from the target subject side (object side) to the light receiving surface 25A side (image side) of the image sensor 25. The lens group 15B includes an anti-vibration lens 15B1, a focus lens (not shown), a zoom lens 15B2, and the like. The zoom lens 15B2 is supported by a lens actuator 21 (described later) so as to be movable along the optical axis OA. The anti-vibration lens 15B1 is supported by a lens actuator 17 (described later) so as to be movable in a direction perpendicular to the optical axis OA.

[0040] By increasing the focal length with the zoom lens 15B2, the camera 10 becomes telephoto, thereby narrowing the angle of view (narrowing the imaging range). By decreasing the focal length with the zoom lens 15B2, the camera 10 becomes wide-angle, thereby widening the angle of view (widening the imaging range).

[0041] The optical system 15 may include various lenses (not shown) in addition to the objective lens 15A and the lens group 15B. The optical system 15 may also include an aperture. The positions of the lenses, lens group, and aperture included in the optical system 15 are not limited, and the technology of the present disclosure is valid even if they are located at positions different from those shown in FIG. 4 .

[0042] The vibration-proof lens 15B1 is movable in a direction perpendicular to the optical axis OA, and the zoom lens 15B2 is movable along the optical axis OA.

[0043] The optical system 15 includes lens actuators 17 and 21. The lens actuator 17 applies a force to the vibration-proof lens 15B1 that fluctuates in a direction perpendicular to the optical axis of the vibration-proof lens 15B1. The lens actuator 17 is controlled by an OIS (Optical Image Stabilizer) driver 23. When the lens actuator 17 is driven under the control of the OIS driver 23, the position of the vibration-proof lens 15B1 fluctuates in a direction perpendicular to the optical axis OA.

[0044] The lens actuator 21 applies a force to the zoom lens 15B2 to move it along the optical axis OA of the optical system 15. The lens actuator 21 is controlled by the lens driver 28. When the lens actuator 21 is driven under the control of the lens driver 28, the position of the zoom lens 15B2 moves along the optical axis OA. When the position of the zoom lens 15B2 moves along the optical axis OA, the focal length of the camera 10 changes.

[0045] In addition, if the outline of the captured image is, for example, a rectangle with a short side in the pitch axis PA direction and a long side in the yaw axis YA direction, the angle of view in the pitch axis PA direction is narrower than the angle of view in the yaw axis YA direction and is narrower than the angle of view of the diagonal.

[0046] By the optical system 15 configured in this manner, light representing the imaging target area is focused on the light receiving surface 25A of the imaging element 25, and the imaging target area is imaged by the imaging element 25.

[0047] Incidentally, vibrations imparted to the camera 10 include, if outdoors, vibrations caused by passing cars, vibrations caused by wind, vibrations caused by road construction, etc., and if indoors, vibrations caused by the operation of an air conditioner, vibrations caused by people entering and exiting, etc. Therefore, the camera 10 shakes due to the vibrations imparted to the camera 10 (hereinafter also simply referred to as "vibrations").

[0048] In this embodiment, "shake" refers to a phenomenon in which the image of a target subject on the light-receiving surface 25A of the image sensor 25 of the camera 10 fluctuates due to a change in the positional relationship between the optical axis OA and the light-receiving surface 25A. In other words, "shake" can also be said to be a phenomenon in which the optical image formed on the light-receiving surface 25A fluctuates due to the tilt of the optical axis OA caused by vibrations applied to the camera 10. The fluctuation of the optical axis OA means, for example, that the optical axis OA is tilted with respect to a reference axis (e.g., the optical axis OA before the shake occurs). Hereinafter, shake caused by vibrations will also be simply referred to as "shake."

[0049] Shake is included in the captured image as a noise component and affects the image quality of the captured image. Therefore, in order to remove the noise component included in the captured image due to shake, the camera 10 is equipped with a lens-side shake correction mechanism 29, an image sensor-side shake correction mechanism 45, and an electronic shake correction unit 33, which are used to correct shake.

[0050] The lens-side shake correction mechanism 29 and the image sensor-side shake correction mechanism 45 are mechanical shake correction mechanisms that apply power generated by a drive source such as a motor (for example, a voice coil motor) to a shake correction element (for example, the vibration-proof lens 15B1 and / or the image sensor 25) to move the shake correction element in a direction perpendicular to the optical axis of the imaging optical system, thereby correcting shake.

[0051] Specifically, the lens-side image stabilization mechanism 29 applies power generated by a drive source such as a motor (e.g., a voice coil motor) to the image stabilization lens 15B1, thereby moving the image stabilization lens 15B1 in a direction perpendicular to the optical axis of the imaging optical system, thereby correcting shake. The image sensor-side image stabilization mechanism 45 applies power generated by a drive source such as a motor (e.g., a voice coil motor) to the image sensor 25, thereby moving the image sensor 25 in a direction perpendicular to the optical axis of the imaging optical system, thereby correcting shake. The electronic image stabilization unit 33 corrects shake by performing image processing on the captured image based on the amount of shake. In other words, the image stabilization unit (image stabilization component) performs shake correction mechanically or electronically using a hardware and / or software configuration. Here, mechanical shake correction refers to shake correction achieved by mechanically moving shake correction elements such as the anti-vibration lens 15B1 and / or the image sensor 25 using power generated by a driving source such as a motor (e.g., a voice coil motor), and electronic shake correction refers to shake correction achieved, for example, by image processing performed by a processor.

[0052] As an example, as shown in FIG. 4, the lens-side shake correction mechanism 29 includes an anti-vibration lens 15B1, a lens actuator 17, an OIS driver 23, and a position sensor 39.

[0053] Various well-known methods can be used to correct shake using the lens-side shake correction mechanism 29. In this embodiment, the method used is to correct shake by moving the vibration-proof lens 15B1 based on the amount of shake detected by a shake amount detection sensor 40 (described later). Specifically, shake correction is performed by moving the vibration-proof lens 15B1 in a direction that cancels out the shake and by an amount that cancels out the shake.

[0054] A lens actuator 17 is attached to the vibration-proof lens 15B1. The lens actuator 17 is a shift mechanism equipped with a voice coil motor, and by driving the voice coil motor, the vibration-proof lens 15B1 is moved in a direction perpendicular to the optical axis of the vibration-proof lens 15B1. Note that, although a shift mechanism equipped with a voice coil motor is used as the lens actuator 17 here, the technology of the present disclosure is not limited to this, and other power sources such as a stepping motor or a piezoelectric element may be used instead of the voice coil motor.

[0055] The lens actuator 17 is controlled by an OIS driver 23. When the lens actuator 17 is driven under the control of the OIS driver 23, the position of the vibration-proof lens 15B1 is mechanically changed within a two-dimensional plane perpendicular to the optical axis OA.

[0056] The position sensor 39 detects the current position of the vibration-proof lens 15B1 and outputs a position signal indicating the detected current position. Here, a device including a Hall element is used as an example of the position sensor 39. Here, the current position of the vibration-proof lens 15B1 refers to the current position within a two-dimensional plane of the vibration-proof lens. The vibration-proof lens two-dimensional plane refers to a two-dimensional plane perpendicular to the optical axis of the vibration-proof lens 15B1. Note that in this embodiment, a device including a Hall element is used as an example of the position sensor 39, but the technology of the present disclosure is not limited to this, and a magnetic sensor, a photosensor, or the like may be used instead of the Hall element.

[0057] The lens-side image stabilization mechanism 29 corrects shake by moving the vibration-proof lens 15B1 along at least one of the pitch axis PA direction and the yaw axis YA direction within the range that is actually captured. In other words, the lens-side image stabilization mechanism 29 corrects shake by moving the vibration-proof lens 15B1 within the two-dimensional vibration-proof lens plane by an amount corresponding to the amount of shake.

[0058] The image sensor side shake correction mechanism 45 includes the image sensor 25 , a BIS (Body Image Stabilizer) driver 22 , an image sensor actuator 27 , and a position sensor 47 .

[0059] As with the shake correction method used by the lens-side shake correction mechanism 29, various well-known methods can be used to correct shake using the image sensor-side shake correction mechanism 45. In this embodiment, the shake correction method used is to correct shake by moving the image sensor 25 based on the amount of shake detected by the shake amount detection sensor 40. Specifically, shake correction is performed by moving the image sensor 25 in a direction that cancels out the shake and by an amount that cancels out the shake.

[0060] An imaging element actuator 27 is attached to the imaging element 25. The imaging element actuator 27 is a shift mechanism equipped with a voice coil motor, and by driving the voice coil motor, the imaging element 25 is moved in a direction perpendicular to the optical axis of the vibration-proof lens 15B1. Note that, although a shift mechanism equipped with a voice coil motor is used as the imaging element actuator 27 here, the technology of the present disclosure is not limited to this, and other power sources such as a stepping motor or a piezoelectric element may be applied instead of the voice coil motor.

[0061] The imaging element actuator 27 is controlled by the BIS driver 22. When the imaging element actuator 27 is driven under the control of the BIS driver 22, the position of the imaging element 25 is mechanically moved in a direction perpendicular to the optical axis OA.

[0062] The position sensor 47 detects the current position of the image sensor 25 and outputs a position signal indicating the detected current position. Here, a device including a Hall element is used as an example of the position sensor 47. Here, the current position of the image sensor 25 refers to the current position within a two-dimensional plane of the image sensor. The two-dimensional plane of the image sensor refers to a two-dimensional plane perpendicular to the optical axis of the vibration-proof lens 15B1. Note that in this embodiment, a device including a Hall element is used as an example of the position sensor 47, but the technology of the present disclosure is not limited to this, and a magnetic sensor, a photosensor, or the like may be used instead of the Hall element.

[0063] The camera 10 includes a computer 19, a DSP (Digital Signal Processor) 31, an image memory 32, an electronic shake correction unit 33, a communication I / F 34, a shake amount detection sensor 40, and a UI (User Interface) device 43. The computer 19 includes a memory 35, a storage 36, and a CPU (Central Processing Unit) 37.

[0064] The image sensor 25, DSP 31, image memory 32, electronic shake correction unit 33, communication I / F 34, memory 35, storage 36, CPU 37, shake amount detection sensor 40, and UI device 43 are connected to a bus 38. The OIS driver 23 is also connected to the bus 38. Note that, although one bus is shown as the bus 38 in the example shown in Fig. 4 for convenience of illustration, multiple buses may be used. The bus 38 may be a serial bus or a parallel bus such as a data bus, an address bus, and a control bus.

[0065] The memory 35 temporarily stores various information and is used as a work memory. An example of the memory 35 is a random access memory (RAM), but other types of storage devices may be used. The storage 36 stores various programs for the camera 10. The CPU 37 reads the various programs from the storage 36 and executes them on the memory 35 to control the entire camera 10. Examples of the storage 36 include a flash memory, an SSD, an EEPROM, and an HDD. Furthermore, various types of non-volatile memory, such as a magnetoresistive memory or a ferroelectric memory, may be used instead of or in combination with the flash memory.

[0066] The image sensor 25 is a CMOS (Complementary Metal Oxide Semiconductor) image sensor. The image sensor 25 captures an image of a target subject at a predetermined frame rate under the direction of the CPU 37. The "predetermined frame rate" here refers to, for example, several tens to several hundreds of frames per second. The image sensor 25 itself may incorporate a control device (image sensor control device), in which case the image sensor control device performs detailed control of the image sensor 25 in accordance with the image capture instructions output by the CPU 37. The image sensor 25 may also capture an image of a target subject at a predetermined frame rate under the direction of the DSP 31, in which case the image sensor control device performs detailed control of the image sensor 25 in accordance with the image capture instructions output by the DSP 31. The DSP 31 is sometimes referred to as an ISP (Image Signal Processor).

[0067] The light-receiving surface 25A of the image sensor 25 is formed by a plurality of photosensitive pixels (not shown) arranged in a matrix. In the image sensor 25, each photosensitive pixel is exposed to light, and photoelectric conversion is performed for each photosensitive pixel. The electric charge obtained by photoelectric conversion for each photosensitive pixel is an analog image signal representing the target subject. Here, a plurality of photoelectric conversion elements (e.g., photoelectric conversion elements with color filters) sensitive to visible light are used as the photosensitive pixels. In the image sensor 25, the plurality of photoelectric conversion elements include photoelectric conversion elements sensitive to R (red) light (e.g., photoelectric conversion elements with R filters corresponding to R), photoelectric conversion elements sensitive to G (green) light (e.g., photoelectric conversion elements with G filters corresponding to G), and photoelectric conversion elements sensitive to B (blue) light (e.g., photoelectric conversion elements with B filters corresponding to B). The camera 10 uses these photosensitive pixels to capture images based on visible light (e.g., light on the short wavelength side of approximately 700 nanometers or less). However, this embodiment is not limited to this, and imaging may be performed based on infrared light (e.g., light with a wavelength longer than approximately 700 nanometers). In this case, a plurality of photoelectric conversion elements sensitive to infrared light may be used as the plurality of photosensitive pixels. In particular, for imaging of short-wavelength infrared (SWIR), for example, an InGaAs sensor and / or a type-II quantum well (T2SL; Simulation of Type-II Quantum Well) sensor may be used.

[0068] The image sensor 25 performs signal processing such as A / D (Analog / Digital) conversion on the analog image signal to generate a digital image, which is a digital image signal. The image sensor 25 is connected to the DSP 31 via a bus 38, and outputs the generated digital image to the DSP 31 via the bus 38 in frame units.

[0069] Note that, although a CMOS image sensor is described here as an example of the image sensor 25, the technology of the present disclosure is not limited thereto, and a CCD (Charge Coupled Device) image sensor may also be used as the image sensor 25. In this case, the image sensor 25 is connected to the bus 38 via an AFE (Analog Front End) (not shown) that has a built-in CCD driver. The AFE generates a digital image by performing signal processing such as A / D conversion on the analog image signal obtained by the image sensor 25, and outputs the generated digital image to the DSP 31. The CCD image sensor is driven by a CCD driver built into the AFE. Of course, the CCD driver may also be provided separately.

[0070] The DSP 31 performs various types of digital signal processing on the digital image. The various types of digital signal processing refer to, for example, demosaic processing, noise removal processing, gradation correction processing, color correction processing, etc. The DSP 31 outputs the digital image after digital signal processing to the image memory 32 for each frame. The image memory 32 stores the digital image from the DSP 31.

[0071] The shake amount detection sensor 40 is a device that includes, for example, a gyro sensor, and detects the amount of shake of the camera 10. In other words, the shake amount detection sensor 40 detects the amount of shake in each of a pair of axial directions. The gyro sensor detects the amount of rotational shake around each of the pitch axis PA, the yaw axis YA, and the roll axis RA (an axis parallel to the optical axis OA) (see FIG. 1 ). The shake amount detection sensor 40 detects the amount of shake of the camera 10 by converting the amount of rotational shake around the pitch axis PA and the amount of rotational shake around the yaw axis YA detected by the gyro sensor into the amount of shake in a two-dimensional plane parallel to the pitch axis PA and the yaw axis YA.

[0072] Here, a gyro sensor is used as an example of the shake amount detection sensor 40, but this is merely an example, and the shake amount detection sensor 40 may also be an acceleration sensor. The acceleration sensor detects the amount of shake in a two-dimensional plane parallel to the pitch axis PA and the yaw axis YA. The shake amount detection sensor 40 outputs the detected amount of shake to the CPU 37.

[0073] Furthermore, although an example in which the amount of shake is detected by a physical sensor, namely, the shake amount detection sensor 40, is given here, the technology of the present disclosure is not limited to this. For example, a motion vector obtained by comparing successive captured images stored in the image memory 32 in chronological order may be used as the amount of shake. Furthermore, the amount of shake that is ultimately used may be derived based on the amount of shake detected by the physical sensor and the motion vector obtained by image processing.

[0074] The CPU 37 acquires the amount of shake detected by the shake amount detection sensor 40, and controls the lens-side shake compensation mechanism 29, the image sensor-side shake compensation mechanism 45, and the electronic shake compensation unit 33 based on the acquired amount of shake. The amount of shake detected by the shake amount detection sensor 40 is used for shake compensation by each of the lens-side shake compensation mechanism 29 and the electronic shake compensation unit 33.

[0075] The electronic shake correction unit 33 is a device including an ASIC (Application Specific Integrated Circuit). The electronic shake correction unit 33 corrects shake by performing image processing on the captured image in the image memory 32 based on the amount of shake detected by the shake amount detection sensor 40.

[0076] Note that, although a device including an ASIC is exemplified as the electronic shake correction unit 33 here, the technology of the present disclosure is not limited to this and may be, for example, a device including a field programmable gate array (FPGA) or a programmable logic device (PLD). Furthermore, for example, the electronic shake correction unit 33 may be a device including multiple ASICs, FPGAs, and PLDs. Furthermore, a computer including a CPU, storage, and memory may be employed as the electronic shake correction unit 33. There may be a single CPU or multiple CPUs. Furthermore, the electronic shake correction unit 33 may be realized by a combination of hardware and software configurations.

[0077] The communication I / F 34 is, for example, a network interface, and controls the transmission of various information between the camera 10 and the management device 11 via the network 20. This network 20 is, for example, a wide area network (WAN) such as the Internet, or a local area network (LAN). The communication I / F 34 performs communication between the camera 10 and the management device 11.

[0078] The UI device 43 includes a reception device 43A and a display 43B. The reception device 43A is, for example, a hard key or a touch panel, and receives various instructions from the user. The CPU 37 acquires the various instructions received by the reception device 43A and operates in accordance with the acquired instructions.

[0079] The display 43B displays various information under the control of the CPU 37. Examples of the various information displayed on the display 43B include the contents of various instructions accepted by the accepting device 43A, captured images, and the like.

[0080] <Configuration of Electrical Systems of Swivel Mechanism 16 and Management Device 11> Fig. 5 is a diagram showing an example of the configuration of the electrical systems of the swing mechanism 16 and the management device 11. As shown in Fig. 5 as an example, the swing mechanism 16 includes a yaw axis swing mechanism 71, a pitch axis swing mechanism 72, motors 73 and 74, drivers 75 and 76, and communication I / Fs 79 and 80.

[0081] The yaw axis rotation mechanism 71 rotates the camera 10 in the yaw direction. The motor 73 generates power by being driven under the control of a driver 75. The yaw axis rotation mechanism 71 rotates the camera 10 in the yaw direction by receiving the power generated by the motor 73. The pitch axis rotation mechanism 72 rotates the camera 10 in the pitch direction. The motor 74 generates power by being driven under the control of a driver 76. The pitch axis rotation mechanism 72 rotates the camera 10 in the pitch direction by receiving the power generated by the motor 74.

[0082] The communication I / Fs 79 and 80 are, for example, network interfaces, and control the transmission of various information between the management device 11 and the network 20. The network 20 is, for example, a WAN such as the Internet or a LAN. The communication I / Fs 79 and 80 perform communication between the swivel mechanism 16 and the management device 11.

[0083] 5, the management device 11 includes a display 13a, a secondary storage device 14, a control device 60, a receiving device 62, and communication I / Fs 66, 67, and 68. The control device 60 includes a CPU 60A, a storage 60B, and a memory 60C. The CPU 60A is an example of a processor according to the present invention.

[0084] The accepting device 62, the display 13a, the secondary storage device 14, the CPU 60A, the storage 60B, the memory 60C, and the communication I / F 66 are each connected to a bus 70. In the example shown in Fig. 5, for convenience of illustration, one bus is shown as the bus 70, but multiple buses may be used. The bus 70 may be a serial bus or a parallel bus including a data bus, an address bus, a control bus, etc.

[0085] The memory 60C temporarily stores various information and is used as a work memory. An example of the memory 60C is a RAM, but other types of storage devices may be used. The storage 60B stores various programs for the management device 11 (hereinafter simply referred to as "management device programs").

[0086] The CPU 60A reads out the management device program from the storage 60B and executes the read out management device program on the memory 60C to control the entire management device 11. The management device program includes the control program of the present invention.

[0087] The communication I / F 66 is, for example, a network interface. The communication I / F 66 is communicatively connected to the communication I / F 34 of the camera 10 via the network 20, and controls the transmission of various information between the camera 10 and the communication I / Fs 67 and 68. The communication I / F 67 is communicatively connected to the communication I / F 79 of the rotation mechanism 16 via the network 20, and controls the transmission of various information between the communication I / F 67 and the yaw axis rotation mechanism 71. The communication I / F 68 is communicatively connected to the communication I / F 80 of the rotation mechanism 16 via the network 20, and controls the transmission of various information between the communication I / F 67 and the pitch axis rotation mechanism 72.

[0088] The CPU 60A receives captured images and imaging information from the camera 10 via the communication I / F 66 and the communication I / F 34. The CPU 60A controls the imaging operation of the camera 10 to capture an image of an object via the communication I / F 66 and the communication I / F 34.

[0089] The CPU 60A controls the driver 75 and motor 73 of the turning mechanism 16 via the communication I / F 67 and communication I / F 79, thereby controlling the turning operation of the yaw axis turning mechanism 71. The CPU 60A also controls the driver 76 and motor 74 of the turning mechanism 16 via the communication I / F 68 and communication I / F 80, thereby controlling the turning operation of the pitch axis turning mechanism 72.

[0090] The CPU 60A detects a moving object from a captured image that is output from the camera 10 as captured image data, and acquires first position information related to the position of the moving object in the captured image. The CPU 60A acquires second position information related to the rotation position of the rotation mechanism 16 that rotates the camera 10 from the rotation mechanism 16. The CPU 60A controls the rotation of the rotation mechanism 16 so that the moving object is included in the imaging range of the camera 10, based on the first position information related to the position of the moving object and the second position information related to the rotation position of the rotation mechanism 16. The CPU 60A also detects the moving object from the captured image while the rotation mechanism 16 is rotating. That is, the CPU 60A detects the moving object from the captured image in parallel with controlling the rotation of the rotation mechanism 16.

[0091] The CPU 60A detects moving objects from captured images during a period from the start to the completion of rotation control for rotating the rotation mechanism 16. Completion of rotation control of the rotation mechanism 16 refers to a state in which the change in the captured image due to the rotation control has ended. The CPU 60A detects moving objects from captured images during a period from the output of a control command for controlling the rotation of the rotation mechanism 16 to the output of the next control command. The CPU 60A repeatedly detects moving objects from captured images in parallel with the rotation control of the rotation mechanism 16. The CPU 60A detects moving objects from captured images every time a captured image captured by the camera 10 is acquired.

[0092] The reception device 62 is, for example, the keyboard 13b, the mouse 13c, or the touch panel of the display 13a, and receives various instructions from the user. The CPU 60A acquires the various instructions received by the reception device 62 and operates in accordance with the acquired instructions. For example, when the reception device 62 receives processing content for the camera 10 and / or the rotation mechanism 16, the CPU 60A operates the camera 10 and / or the rotation mechanism 16 in accordance with the instruction content received by the reception device 62.

[0093] The display 13a displays various types of information under the control of the CPU 60A. Examples of the various types of information displayed on the display 13a include the contents of various instructions accepted by the acceptance device 62, and captured images and imaging information received by the communication I / F 66. The CPU 60A causes the display 13a to display the contents of various instructions accepted by the acceptance device 62, and captured images and imaging information received by the communication I / F 66.

[0094] The secondary storage device 14 is, for example, a non-volatile memory, and stores various types of information under the control of the CPU 60 A. Examples of the various types of information stored in the secondary storage device 14 include captured images and imaging information received by the communication I / F 66. The CPU 60 A stores the captured images and imaging information received by the communication I / F 66 in the secondary storage device 14.

[0095] 6 is a diagram showing, for reference, an example of management processing in a conventional imaging system in which a CPU 60A of the management device 11 controls the rotation of the rotation mechanism 16 to monitor a moving object based on an image captured by the camera 10. As described above, the camera 10 and the rotation mechanism 16 are connected to the management device 11 via the network 20 so as to be able to communicate with each other.

[0096] Camera 10 is mounted on a swivel mechanism 16 and is set to face a predetermined imaging area that is the imaging target. For example, the zoom position of the zoom lens of camera 10 is set to the wide-angle end. As shown in Figure 6, data of images captured by camera 10 is recorded frame by frame in memory 35 of computer 19 mounted on camera 10 and transmitted to management device 11 via network 20.

[0097] The transmitted captured image of camera 10 is displayed, for example, on display 13a of management device 11. When CPU 60A of management device 11 detects a moving object in the captured image while the captured image of camera 10 is being displayed on display 13a, CPU 60A of management device 11 starts a moving object tracking monitoring process to track the detected moving object as a monitoring target.

[0098] Also, for example, a situation may occur in which a user (administrator) is standing in front of the management device 11 and is viewing an image captured by the camera 10 displayed on the display 13a. While the image captured by the camera 10 is being displayed on the display 13a, the CPU 60A of the management device 11 may, for example, start a tracking monitoring process to track an image capture target (e.g., a moving object) designated in response to a designation operation of the user for the image capture target. Specifically, when a moving object is captured in an image displayed on the display 13a and the user performs a touch operation on the moving object, the CPU 60A may start a tracking monitoring process to track the moving object designated by the touch operation as a moving object to be monitored.

[0099] The CPU 60A captures the captured image data transmitted from the camera 10 and records the captured image data in the memory 60C or the secondary storage device 14 mounted on the control device 60. The CPU 60A also performs moving object detection 111a to detect a moving object in the captured image based on the captured image for which the captured image data is output, for example, the captured image output by the captured image data captured by capture 110a.

[0100] The CPU 60A acquires mobile object position information relating to the position of a mobile object detected by the mobile object detection unit 111a. For example, the CPU 60A can calculate the relationship between the coordinates of each position (point) on the captured image displayed on the display 13a and the pan / tilt values ​​of the rotation mechanism 16 corresponding to those coordinates, based on the size and positional relationship between the captured image and each position (point). The calculated pan / tilt values ​​are associated with the coordinates of each position (point) and stored as correspondence information in the memory 60C or the secondary storage device 14. The CPU 60A acquires the pan / tilt values ​​corresponding to the specified position of the mobile object as mobile object position information, based on the correspondence information calculated in advance.

[0101] The CPU 60A acquires rotation mechanism position information relating to the current rotation position of the rotation mechanism 16 that rotates the camera 10 while capturing an image of a moving object. The CPU 60A acquires the current pan / tilt values ​​of the rotation mechanism 16 as the rotation mechanism position information.

[0102] First, CPU 60A controls the rotation of rotation mechanism 16 based on the moving object position information and the rotation mechanism position information so that the moving object detected by moving object detection 111a is displayed in the central region of the captured image captured by camera 10. CPU 60A transmits information such as pan and tilt values ​​of rotation mechanism 16 for displaying the moving object in the central region of the captured image to rotation mechanism 16 via network 20 as control command 121a for rotation mechanism 16.

[0103] The turning mechanism 16 starts a turning operation 161a based on a control command 121a sent from the CPU 60A of the management device 11, and turns the orientation of the camera 10 in the commanded direction.

[0104] When the rotation operation 161a of the rotation mechanism 16 is started, the CPU 60A performs a process of waiting for the rotation mechanism to stop 122a, which waits until the rotation mechanism 16 has completed rotation to the specified rotation position (pan / tilt value) that has been instructed.

[0105] Furthermore, when the rotation operation 161a of the rotation mechanism 16 is completed and the process of waiting for the rotation mechanism to stop 122a is finished, the CPU 60A performs a process of waiting for a video delay 123a for the end of a video delay caused by the captured image being transmitted via the network 20. The video delay time is set depending on the environment of the network 20, etc.

[0106] When the video delay ends and the video delay wait 123a process is completed, the CPU 60A performs the next moving object detection 111b to detect a moving object in the captured image. The moving object detection 111b starts, for example, after a predetermined time (e.g., 300 milliseconds) has elapsed since the previous moving object detection 111a. The captured image used for the moving object detection 111b is, for example, an image output from the captured image data captured in the most recent capture of captured image data at the point in time when the predetermined time has elapsed since the previous moving object detection 111a. In the example shown in the figure, the captured image used for the moving object detection 111b is an image output from the captured image data captured in capture 110j.

[0107] The CPU 60A acquires moving object position information relating to the position of the moving object detected by the moving object detection unit 111b. The process of acquiring the moving object position information is the same as the process of acquiring the first position information of the moving object from the captured image acquired by the acquisition unit 110a.

[0108] The CPU 60A acquires rotation mechanism position information (pan / tilt values) relating to the current rotation position of the rotation mechanism 16 that rotates the camera 10.

[0109] Based on the acquired moving object position information and the acquired turning mechanism position information, CPU 60A controls the turning of turning mechanism 16 so that the moving object detected by moving object detection 111b is displayed in the central region of the captured image captured by camera 10. CPU 60A transmits information such as pan and tilt values ​​of turning mechanism 16 for displaying the moving object in the central region of the captured image to turning mechanism 16 via network 20 as control command 121b for turning mechanism 16.

[0110] The rotation mechanism 16 starts a rotation operation 161b based on a control command 121b sent from the management device 11, and rotates the orientation of the camera 10 in the commanded direction.

[0111] When the rotation operation 161b of the rotation mechanism 16 is started, the CPU 60A performs a rotation mechanism stop waiting process 122b to wait until the rotation mechanism 16 has completed rotation to the specified rotation position (pan / tilt value) that has been instructed.

[0112] In a conventional imaging system, for example, such a process is repeated to control the rotation of the rotation mechanism 16, thereby performing a management process for monitoring a moving object. However, in a process in which the next moving object detection 111b is started after a predetermined time has elapsed since the previous moving object detection 111a, as in the conventional imaging system, the next control command 121b is often not output immediately even after the rotation control based on the previous control command 121a is completed. Therefore, for example, if the moving object is moving at a high speed, the moving object may move out of the angle of view of the camera 10 by the time of the next moving object detection 111b, making it impossible to accurately monitor the moving object.

[0113] <Processing of Management Device in Imaging System 1 of This Embodiment> In contrast, the imaging system 1 of this embodiment performs the following management process to capture images of moving objects without departing from the angle of view of the camera 10. FIG. 7 is a diagram showing an example of the management process performed by the imaging system 1 of this embodiment. As shown in FIG. 7, in the imaging system 1 of this embodiment, the CPU 60A of the management device 11 executes moving object detection 111a to 111n on the captured images captured from the camera 10 by the captures 110a to 110n. The CPU 60A acquires moving object position information relating to the positions of the moving objects detected in the moving object detection 111a to 111n. The method of acquiring moving object position information is the same as the method of acquiring moving object position information in conventional imaging systems. The moving object position information is an example of first position information in the present invention.

[0114] The CPU 60A also executes predictions 131a-131l of the moving positions of the moving objects based on the moving object position information detected by each of the moving object detections 111a-111n, and sequentially acquires moving object position information related to the positions of the moving objects. For example, the CPU 60A calculates the movement (speed and direction) of the moving object based on the moving object position information detected by the moving object detection 111a and the moving object position information detected by the moving object detection 111b, and executes prediction 131a to acquire moving object position information of the moving object at a time point prior to the next moving object detection 111c. The CPU 60A also executes prediction 131b to calculate the movement (speed and direction) of the moving object based on the moving object position information predicted by prediction 131a and the moving object position information detected by the moving object detection 111c, and acquires moving object position information of the moving object at a time point prior to the next moving object detection 111d. Similarly, prediction 131c is performed based on the mobile object position information predicted in prediction 131b and the mobile object position information detected in mobile object detection 111d, and mobile object position information of the mobile object at a time before the next mobile object detection 111e is obtained.

[0115] The CPU 60A acquires rotation mechanism position information (pan / tilt values) relating to the current rotation position of the rotation mechanism 16 that rotates the camera 10. The rotation mechanism position information is an example of second position information in the present invention.

[0116] First, CPU 60A controls the rotation of rotation mechanism 16 based on the moving body position information acquired by moving body detection 111a and the current rotation mechanism position information, so that the moving body detected by moving body detection 111a is displayed in the central region of the captured image captured by camera 10. CPU 60A transmits information such as pan / tilt values ​​of rotation mechanism 16 for displaying the moving body in the central region of the captured image to rotation mechanism 16 via network 20 as control command 121a for rotation mechanism 16.

[0117] The rotation mechanism 16 starts a rotation operation 161a based on a control command 121a transmitted from the management device 11, and rotates the orientation of the camera 10 in the commanded direction. In the imaging system 1 of this embodiment, moving object detection 111b to 111i for the captured images acquired from the camera 10 is repeatedly performed in parallel with the rotation control of the rotation mechanism 16. For example, moving object detection 111b to 111f for the captured images acquired from the camera 10 is also performed during the rotation operation 161a of the rotation mechanism 16.

[0118] When the rotation operation 161a of the rotation mechanism 16 is initiated, the CPU 60A performs a rotation mechanism stop wait 122a process, which waits until the rotation mechanism 16 has completed rotation to the specified rotation position (pan / tilt value) that was commanded. The end of the rotation mechanism stop wait 122a can be detected based on a rotation operation completion notification signal for the rotation operation 161a transmitted from the rotation mechanism 16. Note that the time required for transmitting the control command from the management device 11 to the rotation mechanism 16 and the time required for transmitting the completion notification signal from the rotation mechanism 16 to the management device 11 are short, and therefore have little impact on the overall processing time of the management device 11. Furthermore, the time required for the rotation mechanism 16 to complete rotation can be calculated, for example, based on the angle and speed at which the rotation mechanism 16 is rotated, and this time may be set as the time for the rotation mechanism stop wait 122a process.

[0119] When the rotation operation 161a of the rotation mechanism 16 is completed and the rotation mechanism stop wait process 122a is completed, the CPU 60A performs a video delay wait process 123a, which waits for the end of the video delay caused by the transmission of the captured image via the network 20. Whether the video delay has ended can be determined by analyzing the captured image captured by the captures 110a to 110n. The CPU 60A makes this determination, for example, based on the background portion (e.g., background feature points appearing in the background of a moving object) captured in the captured image. When the rotation operation 161a of the rotation mechanism 16 stops, the background feature points in the captured image become stationary in response to the stoppage. The CPU 60A determines that the video delay has ended based on the stoppage of the background feature points, and calculates the video delay time from the time when the rotation operation 161a of the rotation mechanism 16 is stopped and the time when the background feature points in the captured image stop. The video delay time is the period from when the rotation of the rotation mechanism 16 is stopped to when the background feature points in the captured image stop.

[0120] When the video delay ends and the processing of the video delay wait 123a is completed, the CPU 60A acquires the mobile object position information of the mobile object predicted in the most recent prediction 131h for the completion time point. Note that the mobile object position information predicted in the prediction 131h is mobile object position information predicted based on the mobile object position information predicted in the prediction 131g and the mobile object position information detected in the mobile object detection 111i.

[0121] The CPU 60A also acquires rotation mechanism position information (pan / tilt values) relating to the current rotation position of the rotation mechanism 16 that rotates the camera 10.

[0122] Based on the moving object position information acquired in prediction 131h and the current rotating mechanism position information, CPU 60A controls the rotation of rotating mechanism 16 so that the imaging range of camera 10 changes toward the predicted moving position of the moving object. CPU 60A transmits information such as pan and tilt values ​​of rotating mechanism 16 for imaging the moving object at the predicted moving position as control command 121b for rotating mechanism 16 to rotating mechanism 16 via network 20. In imaging system 1 of this embodiment, CPU 60A performs moving object detection 111b-111i for the captured image captured from camera 10 between control command 121a for controlling the rotation of rotating mechanism 16 and the next control command 121b. For example, CPU 60A performs moving object detection 111b-111i during the period from the transmission of control command 121a, which is the start of rotation control, to the end of video delay wait 123a, which is the completion of rotation control.

[0123] The rotation mechanism 16 starts a rotation operation 161b based on a control command 121b sent from the management device 11, and rotates the orientation of the camera 10 in the instructed direction so that the moving object being monitored continues to be included in the imaging range of the camera 10, i.e., so that the camera 10 can continue to image the moving object.

[0124] When the rotation operation 161b of the rotation mechanism 16 is started, the CPU 60A performs a rotation mechanism stop waiting process 122b to wait until the rotation mechanism 16 has completed rotation to the specified rotation position (pan / tilt value) that has been instructed.

[0125] As described above, the CPU 60A of the management device 11 performs moving object detection 111a-111n on each captured image captured by the camera 10. This allows the movement path of the moving object to be detected in detail, and the moving object can be captured without straying from the imaging range of the camera 10. Furthermore, the CPU 60A repeatedly performs moving object detection 111a-111n on the captured image in parallel with the rotation control of the rotation mechanism 16. This allows the movement of the moving object to be accurately recognized, and the moving object can be captured without straying from the imaging range of the camera 10.

[0126] <Movement Prediction Based on Movement Vector of Moving Object> Fig. 8 is a diagram showing an example of predicting the movement position of a moving object based on the movement vector of the moving object in the management device 11 of the imaging system 1. As shown in Fig. 8, the CPU 60A of the management device 11 performs moving object detection 111a on the captured image of the camera 10 captured in capture 110a.

[0127] The CPU 60A controls the rotation of the rotation mechanism 16 based on the moving object position information acquired by the moving object detection 111a and the current rotation mechanism position information so that the moving object detected by the moving object detection 111a is displayed in the central region of the captured image captured by the camera 10. Specifically, the CPU 60A transmits pan / tilt values ​​of the rotation mechanism 16 for displaying the moving object in the central region of the captured image as a control command 121a to the rotation mechanism 16. When the rotation operation 161a of the rotation mechanism 16 based on the control command 121a is started, the CPU 60A performs processing of waiting for the rotation mechanism to stop 122a until the rotation operation 161a is completed.

[0128] The CPU 60A performs a process of estimating a rotation movement vector 171a of the rotation mechanism 16 based on the captured image of the camera 10 captured in capture 110a and the captured image of the camera 10 captured in capture 110b, and acquires a rotation movement vector. The rotation movement vector is a vector that indicates the movement of the imaging range of the camera 10 due to the rotation control of the rotation mechanism 16. The CPU 60A acquires the rotation movement vector from the captured image of the camera 10. For example, the CPU 60A acquires the rotation movement vector based on feature points such as background portions in the captured image. The rotation movement vector is an example of a second movement vector in the present invention.

[0129] The CPU 60A performs moving object detection 111b on the captured image of the camera 10 captured in capture 110b. Based on the moving object position information detected in moving object detection 111b and the moving object position information detected in moving object detection 111a, the CPU 60A acquires an on-image movement vector indicating the movement of the moving object in the image within the imaging range of the camera 10. The CPU 60A acquires the on-image movement vector from the captured image of the camera 10. For example, the CPU 60A acquires the on-image movement vector based on feature points of the moving object in the captured image. The on-image movement vector is an example of a first movement vector in the present invention.

[0130] The CPU 60A performs processing of the moving body movement vector estimation 181a of the moving body based on the on-image movement vector acquired by the moving body detection 111b and the turning movement vector acquired by the turning movement vector estimation 171a to acquire the moving body movement vector. The moving body movement vector is a vector that indicates the movement of the moving body in the space in which the moving body exists. The moving body movement vector is an example of the third movement vector in the present invention.

[0131] The CPU 60A calculates the movement (speed and direction) of the moving object based on the moving object movement vector acquired by the moving object movement vector estimation 181a and the moving object position information detected by the moving object detection 111a, executes prediction 131a, and acquires the moving object position information of the moving object. The CPU 60A also executes prediction 131b based on the moving object movement vector acquired by the moving object movement vector estimation 181b and the moving object position information acquired by prediction 131a, and acquires the moving object position information of the moving object. Thereafter, similarly, the CPU 60A sequentially acquires the moving object position information of the moving object, and outputs control command 121b for controlling the rotation of the rotation mechanism 16 based on the moving object position information acquired by prediction 131h, for example, as in the process of FIG. 7. The moving object position information acquired by predictions 131a to 131d based on the moving object movement vector (third movement vector) is an example of the third position information of the present invention.

[0132] Although the notation of the video delay wait time is omitted in Figure 8, for example, the video delay wait time may be estimated based on the relationship between information regarding the rotation control of the rotation mechanism 16 and the rotation movement vector obtained in the process of estimating the rotation movement vector of the rotation mechanism 16.

[0133] In this example, the CPU 60A of the management device 11 performs a process of estimating the moving object's moving vector based on the on-image moving vector (first moving vector) acquired by moving object detection and the turning moving vector (second moving vector) acquired by turning moving vector estimation, and acquires the moving object's moving vector (third moving vector). With this configuration, it is possible to appropriately acquire the moving object's moving vector in the space in which the moving object exists, and to capture an image of the moving object so that it does not stray from the imaging range of the camera 10.

[0134] <Rotation Control Based on Feature Points Determined by History Information> Figure 9 is a diagram showing an example of predicting the movement position of a moving object based on feature points determined by history information in the management device 11 of the imaging system 1. As shown in Figure 9, the CPU 60A of the management device 11 performs moving object detection 111a on the captured image of the camera 10 acquired by acquisition 110a. The CPU 60A performs rotation control of the rotation mechanism 16 based on the moving object position information acquired by the moving object detection 111a and the current rotation mechanism position information so that the moving object detected by the moving object detection 111a is displayed in the central region of the captured image captured by the camera 10.

[0135] The CPU 60A transmits pan / tilt values ​​of the rotation mechanism 16 for displaying the moving object in the central region of the captured image as the control command 121a to the rotation mechanism 16. For example, when the moving object is detected in the lower right direction relative to the central region of the captured image when viewed from the front of the display 13a, the CPU 60A transmits pan / tilt values ​​for rotating the rotation mechanism 16 in the lower right direction as the control command 121a to the rotation mechanism 16. The "lower right direction" for rotating the rotation mechanism 16 is an example of history information in the present invention.

[0136] 8, CPU 60A performs processing of rotation movement vector estimation 171a of rotation mechanism 16 based on the captured image of camera 10 captured in capture 110a and the captured image of camera 10 captured in capture 110b to obtain a rotation movement vector (second movement vector). Similarly, CPU 60A performs processing of rotation movement vector estimation 171b, 171c, and 171d to obtain the respective rotation movement vectors.

[0137] The CPU 60A may acquire the turning movement vector based on the control command 121a and history information of the turning control. That is, if the current control command has been issued in the past, the turning movement vector can be acquired from history information of the past control commands. In this way, by using the history information of the turning control, it is possible to acquire the turning movement vector without using captured images.

[0138] For example, among the turning movement vectors acquired by turning movement vector estimations 171a to 171d, as shown in Figure 9, the turning movement vectors acquired by turning movement vector estimations 171a and 171b are turning movement vectors with no movement (zero vectors), and the turning movement vectors acquired by turning movement vector estimations 171c and 171d are turning movement vectors in the lower right direction.

[0139] In this case, CPU 60A uses the turning movement vector that is consistent with the downward right direction, which is the turning direction of turning mechanism 16 transmitted by control command 121a, in the processing of moving body movement vector estimation, and does not use the turning movement vector that does not match, in the processing of moving body movement vector estimation. In other words, CPU 60A uses the turning movement vector in the downward right direction acquired by turning movement vector estimations 171c and 171d in the processing of moving body movement vector estimation, and does not use the turning movement vector without movement acquired by turning movement vector estimations 171c and 171d in the processing of moving body movement vector estimation.

[0140] In this example, the CPU 60A of the management device 11 uses a rotation movement vector that is consistent with historical information related to the rotation control of the rotation mechanism 16 in the process of estimating the moving object movement vector, and does not use a rotation movement vector that is inconsistent in the process of estimating the moving object movement vector. With this configuration, it is possible to appropriately acquire a rotation movement vector in the process of estimating the moving object movement vector, and it is possible to capture an image of the moving object so that it does not deviate from the imaging range of the camera 10.

[0141] <Modification of the processing of the management device in the imaging system 1 of this embodiment> Figure 10 is a diagram showing a modification of the management processing by the imaging system 1 of this embodiment. As shown in Figure 10, in the case of the modification of the management processing, the CPU 60A of the management device 11 executes moving object detection 111a to 111n on the captured images captured from the camera 10 by each of capture 110a to 110n, and acquires moving object position information relating to the position of the moving object detected by each of the moving object detection 111a to 111n, which is the same as the management processing described above in Figure 7.

[0142] Also, similar to the management process described above in FIG. 7, the CPU 60A executes predictions 131a to 131l of the moving position of the moving body based on the moving body position information detected by each moving body detection 111a to 111n, and sequentially acquires moving body position information regarding the position of the moving body.

[0143] 7. Also, the CPU 60A acquires rotation mechanism position information (pan / tilt values) relating to the current rotation position of the rotation mechanism 16, similarly to the management processing described above with reference to FIG.

[0144] Then, first, the CPU 60A controls the rotation of the rotation mechanism 16 based on the moving body position information and the rotation mechanism position information so that the moving body detected by the moving body detection 111a is displayed in the central area of ​​the captured image, and sends a control command 121a to the rotation mechanism 16, which is the same as the management process described above in Figure 7.

[0145] Also, the rotation mechanism 16 starts a rotation operation 161a based on a control command 121a sent from the CPU 60A of the management device 11, and rotates the camera 10 in the commanded direction, which is similar to the management process described above in Figure 7.

[0146] Also, similar to the management process described above in FIG. 7, when the rotation operation 161a of the rotation mechanism 16 is started, the CPU 60A performs a process of waiting for the rotation mechanism to stop 122a, which waits until the rotation mechanism 16 has completed rotation to the specified rotation position (pan / tilt value) that was instructed.

[0147] However, in the modified management process, the CPU 60A does not execute the video delay wait 123a process, which waits for the end of the video delay caused by the captured image being transmitted via the network 20, after the rotation operation 161a of the rotation mechanism 16 is completed and the process of waiting for the rotation mechanism to stop 122a is finished, which is different from the management process that executes the video delay wait 123a process described above in Figure 7.

[0148] When the rotation operation 161a of the rotation mechanism 16 is completed and the process of waiting for the rotation mechanism to stop 122a is finished, the CPU 60A acquires the mobile object position information of the mobile object predicted by the most recent prediction 131e for the end point. Note that the mobile object position information predicted by the prediction 131e is mobile object position information predicted based on the mobile object position information predicted by the prediction 131d and the mobile object position information detected by the mobile object detection 111f.

[0149] The CPU 60A also acquires rotation mechanism position information (pan / tilt values) relating to the current rotation position of the rotation mechanism 16 that rotates the camera 10.

[0150] Based on the moving object position information acquired in prediction 131e and the current rotating mechanism position information, CPU 60A controls the rotation of rotating mechanism 16 so that the imaging range of camera 10 changes toward the predicted moving position of the moving object. CPU 60A transmits information such as pan and tilt values ​​of rotating mechanism 16 for imaging the moving object at the predicted moving position to rotating mechanism 16 via network 20 as control command 121b for rotating mechanism 16.

[0151] In the modified example of the management process, the CPU 60A executes the rotation control of the rotation mechanism 16 based on the control command 121b before the captured image finishes changing during the rotation control of the rotation mechanism 16 based on the control command 121a (i.e., the image delay waiting process is not executed). The rotation control of the rotation mechanism 16 based on the control command 121a is an example of the first rotation control of the present invention. The rotation control of the rotation mechanism 16 based on the control command 121b is an example of the second rotation control of the present invention.

[0152] 7 in that the rotation mechanism 16 starts rotation operation 161b based on control command 121b, and rotates the orientation of camera 10 in the commanded direction so that the predetermined moving object continues to be included in the imaging range of camera 10, i.e., so that the predetermined moving object can continue to be imaged by camera 10. Also, when rotation operation 161b of rotation mechanism 16 starts, CPU 60A performs processing 122b to wait for the rotation mechanism to stop, which is also the same as the management processing described above in FIG.

[0153] In this modification, the CPU 60A of the management device 11 outputs the next control command 121b after completing the process of waiting for the rotation mechanism to stop 122a based on the previous control command 121a, without executing the process of waiting for the video delay 123a to wait for the video delay to end. This configuration improves the tracking ability of the camera 10 with respect to the moving object, making it easier to prevent the moving object from moving out of the imaging range of the camera 10.

[0154] <Modified Swing Control Based on Movement Vector> The swing control (second swing control) of the swing mechanism 16 based on the control command of the modified example described above will be explained below using the movement vector of the moving body.

[0155] The CPU 60A acquires a future rotation movement vector indicating the movement of the imaging range of the camera 10 based on the control value in the future rotation control of the rotation mechanism 16. The future rotation movement vector is the amount of movement of the imaging range planned in the future rotation control. The CPU 60A also performs a process of estimating the rotation movement vector of the rotation mechanism 16 based on the captured image captured from the camera 10, and acquires a next rotation movement vector indicating the movement of the imaging range of the camera 10. The next rotation movement vector is the actual amount of movement of the imaging range. The CPU 60A performs the next rotation control of the rotation mechanism 16 based on the future rotation movement vector and the next rotation movement vector. The future rotation control of the rotation mechanism 16 is an example of the first rotation control of the present invention. The next rotation control of the rotation mechanism 16 is an example of the second rotation control of the present invention. The future rotation movement vector is an example of the fourth movement vector of the present invention. The next rotation movement vector is an example of the fifth movement vector of the present invention.

[0156] As described above, in the modified example, when the process of waiting for the rotation mechanism to stop is completed, CPU 60A acquires the predicted moving body position information of the moving body without executing the process of waiting for the image to be delayed, and performs the next rotation control of rotation mechanism 16. Therefore, on the captured image captured in management device 11, of the previous rotation movement vector in the previous rotation control, the rotation movement vector has moved by an amount corresponding to the period of waiting for the rotation mechanism to stop, but the rotation movement vector corresponding to the period of waiting for the image to be delayed has not yet moved completely.

[0157] Therefore, although the rotation of the rotation mechanism 16 has actually ended, the captured image displayed on the management device 11 still shows the background (rotation mechanism 16) moving due to the video delay. Therefore, the CPU 60A acquires the next movement vector by taking into account (subtracting) the portion of the previous movement vector in the previous rotation control that has not yet been fully moved, and performs the next rotation control based on the subtracted next movement vector. This makes it possible to appropriately acquire the movement vector for the next rotation control, and to capture an image of the moving object so that it does not deviate from the imaging range of the camera 10.

[0158] A modified example of the control by the CPU 60A will be described below.

[0159] <First Modification> CPU 60A may determine the rotation speed at which rotation mechanism 16 is rotated in accordance with, for example, the image capturing conditions of camera 10. The image capturing conditions of camera 10 are, for example, the shutter speed of camera 10. Specifically, CPU 60A controls the rotation operation of rotation mechanism 16 so that the rotation speed does not exceed an upper limit value of the rotation speed of rotation mechanism 16 that is set in accordance with the shutter speed of camera 10.

[0160] This makes it possible to suppress blurring of the captured image, for example, even when the shutter speed of the camera 10 is increased, and to appropriately extract feature points in the captured image and detect moving objects. Furthermore, the CPU 60A may stop detecting moving objects in the captured image when the rotation speed of the rotation mechanism 16 becomes equal to or greater than a predetermined speed relative to the shutter speed of the camera 10. This makes it possible to stop extracting feature points and detecting moving objects in blurred captured images, thereby reducing the computing resources of the device.

[0161] 7, the management device executes predictions 131a to 131l of the moving positions of the moving objects based on the moving object position information detected by the moving object detection devices 111a to 111n, but the present invention is not limited to this. In FIG. 7, the moving object position information of the moving object used to output the control command 121b is position information based on prediction 131h.

[0162] Therefore, for example, even if the predictions 131a to 131g do not predict the mobile object position information, if the mobile object position information is acquired by the mobile object detections 111a to 111i, the prediction 131h can predict the mobile object position information based on that position information. Therefore, the CPU 60A may omit the processes of the predictions 131a to 131g in the process of predicting the moving position of the mobile object in Figure 7. This can improve the processing efficiency of the CPU 60A of the management device 11.

[0163] <Third Modification> The CPU 60A may change the imaging range (zoom) of the camera 10 depending on the length of the video delay wait time. For example, the CPU 60A may change the zoom of the camera 10 to the telephoto side when the video delay wait time is long. The CPU 60A may also change the imaging range (zoom) of the camera 10 depending on the size of the moving object movement vector of the moving object. For example, the CPU 60A may change the zoom of the camera 10 to the telephoto side when the moving object movement vector is large. This makes it less likely that the moving object to be imaged will deviate from the imaging range of the camera 10.

[0164] 8, the rotational movement vector of the rotation mechanism 16 is obtained based on feature points such as the background of the moving object in the captured image. However, if the background that serves as a feature point cannot be detected in the captured image, for example, if the background of the moving object is a blue sky, it may not be possible to determine the rotational movement vector of the rotation mechanism 16 based on the feature points.

[0165] In this case, the CPU 60A may change the processing content of the rotation control depending on whether or not the rotation movement vector can be obtained. For example, the CPU 60A may execute rotation control that performs image delay waiting processing when the rotation movement vector can be obtained, and may switch to execute rotation control that does not perform image delay waiting processing when the rotation movement vector cannot be obtained. This makes it less likely that the moving object to be captured will deviate from the imaging range of the camera 10.

[0166] 8, the CPU 60A obtains the moving object's moving vector based on the on-image moving object vector of the moving object obtained by moving object detection and the rotational moving vector of the rotation mechanism 16 obtained by rotational moving object vector estimation, and predicts the moving position of the moving object based on the moving object moving vector. However, the magnitude of the moving object's moving vector is not necessarily constant and can change depending on the surrounding conditions. Therefore, the magnitudes of the on-image moving object vector and the moving object moving vector often change.

[0167] Therefore, when the randomness of the moving body's movement vector is small, for example, when the moving body's movement is close to uniform motion, the CPU 60A predicts the moving body's movement position based on the acquired moving body movement vector. On the other hand, when the randomness of the moving body's movement vector is large, the CPU 60A may predict the moving body's movement position by reducing the reference ratio of the acquired moving body movement vector. This improves the predictability of the moving body's movement position based on the moving body movement vector.

[0168] <Storage medium for information processing program> In each of the above management controls, an example has been given in which the information processing program of each embodiment is stored in the storage 60B of the management device 11, and the CPU 60A of the management device 11 executes the information processing program in the memory 60C, but the technology of the present disclosure is not limited to this.

[0169] 11 is a diagram showing an example of how an information processing program for management control is installed from a storage medium storing the information processing program into the control device 60 of the management device 11. As an example, as shown in FIG. 11, the information processing program 221 may be stored in a storage medium 220, which is a non-transitory storage medium. In the example shown in FIG. 11, the information processing program 221 stored in the storage medium 220 is installed into the control device 60, and the CPU 60A executes the above-described processes in accordance with the information processing program 221.

[0170] Although various embodiments have been described above, it goes without saying that the present invention is not limited to such examples. It is clear that a person skilled in the art can conceive of various modifications or alterations within the scope of the claims, and it is understood that these also naturally fall within the technical scope of the present invention. Furthermore, the components of the above embodiments may be combined in any manner as long as they do not deviate from the spirit of the invention.

[0171] This application is based on a Japanese patent application (Patent Application No. 2022-147975) filed on September 16, 2022, the contents of which are incorporated herein by reference.

[0172] 1 Imaging system 10 Camera 11 Management device 13a, 43B Display 13b Keyboard 13c Mouse 14 Secondary storage device 15 Optical system 15B Lens group 15B1 Anti-vibration lens 15B2 Zoom lens 16 Swivel mechanism 17, 21 Lens actuator 19 Computer 20 Network 22, 23, 75, 76 Driver 22 BIS driver 23 OIS driver 25 Imaging element 25A Light receiving surface 27 Imaging element actuator 28 Lens driver 29, 45 Correction mechanism 31 DSP 32 Image memory 33 Correction unit 34, 66 to 68, 79, 80 Communication I / F 35, 60C Memory 36, 60B Storage 37, 60A CPU 38, 70 Bus 39, 47 Position sensor 40 Quantity detection sensor 43 System device 43 UI system device 43A, 62 Reception device 60 Control device 71 Yaw axis rotation mechanism 72 Pitch axis rotation mechanism 73, 74 Motor 110a to 110n Capture 111a to 111n Moving body detection 121a, 121b Control command 122b Stop waiting process 131a to 131l Prediction 161a, 161b Rotation operation 171a to 171d Rotation movement vector estimation 181a, 181b Moving body movement vector estimation 220 Storage medium 221 Information processing program

Claims

1. A control device including a processor that controls an imaging device and a rotation device that rotates the imaging device, The processor, Detecting a moving object from the captured image obtained by outputting the captured image data acquired from the imaging device; acquiring first position information relating to a position of the moving object in the captured image; Obtaining second position information relating to a rotation position of the rotation device; performing a rotation control of the rotation device based on the first position information and the second position information so that the moving object is included in an imaging range of the imaging device; the detection and the prediction of the moving position of the moving object based on the detection are repeatedly performed even during a rotation operation of the rotation device; In the turning control, the next control command is output after the turning operation started based on the control command is completed, and the next control command for starting the next turning operation is determined based on the most recent prediction of the time when the next turning operation will be started. Control device.

2. The control device according to claim 1 , The detection is performed in parallel with the turning control. Control device.

3. The control device according to claim 1 , The imaging device performs imaging control regarding the imaging range. Control device.

4. The control device according to claim 1 , The detection is performed during a period including a start to a completion of the turning control. Control device.

5. The control device according to claim 4, The completion of the rotation control is a state in which the change in the captured image due to the rotation control is completed. Control device.

6. The control device according to claim 1 , The detection is performed during a period between control commands of the turning control. Control device.

7. A control device as claimed in claim 1, the detection is performed every time the captured image data is acquired by the imaging device; Control device.

8. A control device as claimed in claim 1, The processor, acquiring a third movement vector which is a movement vector of the moving body in a space in which the moving body exists, based on a first movement vector which is a movement vector of the moving body in the imaging range and a second movement vector which is a movement vector of the imaging range by the turning control; obtaining third position information relating to a position of the moving object based on the third motion vector; performing a rotation control of the rotation device based on the third position information; Control device.

9. The control device according to claim 8, The processor, acquiring the first movement vector and the second movement vector from the captured image; Control device.

10. The control device according to claim 9, The processor, acquiring the first movement vector and the second movement vector based on a feature point of the captured image; Control device.

11. The control device according to claim 10, The processor, acquiring the second movement vector based on the characteristic point determined based on history information of the turning control; Control device.

12. The control device according to claim 8, The processor, obtaining the second movement vector using a control command for the turning control and history information for the turning control; Control device.

13. A control device as claimed in claim 1, The processor, performing a second rotation control of the rotation device before the change in the captured image due to the first rotation control of the rotation device is completed; Control device.

14. The control device according to claim 13, The processor, The second turning control is performed based on a fourth movement vector of the imaging range based on a control value of the first turning control and a fifth movement vector of the imaging range based on the captured image. Control device.

15. The control device according to claim 1, The processor, determining a rotation speed of the rotation device according to an imaging condition of the imaging device; Control device.

16. A control device according to any one of claims 1 to 15, The imaging device and the rotation device are connected via a network. Control device.

17. An imaging device; a rotation device that rotates the imaging device; a control device for controlling the imaging device and the rotation device, The control device includes: Detecting a moving object from the captured image obtained by outputting the captured image data acquired from the imaging device; acquiring first position information relating to a position of the moving object in the captured image; Obtaining second position information relating to a rotation position of the rotation device; performing a rotation control of the rotation device based on the first position information and the second position information so that the moving object is included in an imaging range of the imaging device; the detection and the prediction of the moving position of the moving object based on the detection are repeatedly performed even during a turning operation of the turning device; In the turning control, the next control command is output after the turning operation started based on the control command is completed, and the next control command for starting the next turning operation is determined based on the most recent prediction of the time when the next turning operation will be started. Imaging system.

18. A control method using a control device having a processor that controls an imaging device and a rotation device that rotates the imaging device, comprising: The processor, Detecting a moving object from the captured image obtained by outputting the captured image data acquired from the imaging device; acquiring first position information relating to a position of the moving object in the captured image; Obtaining second position information relating to a rotation position of the rotation device; performing a rotation control of the rotation device based on the first position information and the second position information so that the moving object is included in an imaging range of the imaging device; the detection and the prediction of the moving position of the moving object based on the detection are repeatedly performed even during a rotation operation of the rotation device; In the turning control, the next control command is output after the turning operation started based on the control command is completed, and the next control command for starting the next turning operation is determined based on the most recent prediction of the time when the next turning operation will be started. Control methods.

19. A control program for a control device having a processor that controls an imaging device and a rotation device that rotates the imaging device, The processor, Detecting a moving object from the captured image obtained by outputting the captured image data acquired from the imaging device; acquiring first position information relating to a position of the moving object in the captured image; Obtaining second position information relating to a rotation position of the rotation device; performing a rotation control of the rotation device based on the first position information and the second position information so that the moving object is included in an imaging range of the imaging device; the detection and the prediction of the moving position of the moving object based on the detection are repeatedly performed even during a rotation operation of the rotation device; In the turning control, the next control command is output after the turning operation started based on the control command is completed, and the next control command for starting the next turning operation is determined based on the most recent prediction of the time when the next turning operation will be started. A control program for executing processing.