Image stabilizer, imaging device, monitoring system, and program

Abstract

To provide a shake correction device that eliminates disturbances appearing in an image without relying on a mechanical shake correction unit.SOLUTION: In a monitoring camera which is an imaging device, a shake correction device comprises: a shake amount detection sensor 40; a mechanical shake correction unit 29; an electronic shake correction unit 33 that performs image processing on the image obtained by being imaged by the imaging device, on the basis of operation state information regarding the operation state of an optical element in the imaging device during exposure and a shake amount, so as to correct shake; and a CPU 37 that executes a supplementary shake correction unit to correct shake, by applying, to the image, a filter that is determined according to an operation state information, ratio, shake amount and an exposure period in the imaging device, when the mechanical shake correction unit and the electronic shake correction unit are operated during exposure at a prescribed ratio of sharing.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The technology disclosed herein relates to a blur correction device, an imaging device, a monitoring system, and a program. [Background technology]

[0002] In the following description, "blur" refers to a phenomenon in which, in an imaging device in which subject light indicating a subject is imaged on a light receiving surface via an optical system, the subject image obtained by imaging the subject light on the light receiving surface fluctuates due to a change in the positional relationship between the optical axis of the optical system and the light receiving surface caused by vibrations applied to the imaging device.

[0003] The subject image formed on the imaging plane of the imaging device is roughly divided into an optical image and an electronic image, and the blur of the optical image is corrected by a mechanical correction unit, while the blur of the image is corrected by an electronic correction unit. To correct the blur of the subject image, a blur correction amount derived based on the detected amount of blur is used. The mechanical correction unit corrects the blur by mechanically moving the optical system and / or the imaging element based on the blur correction amount. The electronic correction unit corrects the blur by cropping the captured image according to the blur (see, for example, JP 2013-135442 A, JP 2017-152995 A, and JP 09-163215 A).

[0004] However, electronic blur correction cannot remove the disturbances that appear in an image due to blurring that occurs during exposure. This is because electronic blur correction is a process of cutting out a partial image area from an image, and does not address blurring that occurs during exposure. Known methods for suppressing disturbances that appear in an image due to blurring that occurs during exposure include a method of performing mechanical blur correction during exposure and a method of applying a filter to an image (see, for example, JP 2008-124728 A). Summary of the Invention

[0005] One embodiment of the technology disclosed herein provides a blur correction device, an imaging device, a monitoring system, and a program that can remove disturbances that appear in an image due to blurring that occurs during exposure without relying on a mechanical blur correction unit when blur correction is shared between a mechanical blur correction unit and an electronic blur correction unit. [Means for solving the problem]

[0006] A first aspect of the technology of the present disclosure is a blur correction device including: a detection unit that detects the amount of blur of an imaging device; a mechanical blur correction unit that corrects blur by moving an optical element included in the imaging device based on the amount of blur; an electronic blur correction unit that corrects blur by performing image processing on an image obtained by imaging with the imaging device based on operation status information related to the operation status of the optical element during exposure in the imaging device and the amount of blur; and a supplementary blur correction unit that corrects blur by applying a filter determined according to the operation status information, the ratio, the amount of blur, and the exposure period of the imaging device to the image when the mechanical blur correction unit and the electronic blur correction unit are operated in a shared manner at a predetermined ratio during exposure. As a result, when the mechanical blur correction unit and the electronic blur correction unit are operated in a shared manner to correct blur, disturbances that appear in the image due to blur occurring during exposure can be removed without relying on the mechanical blur correction unit.

[0007] A second aspect of the technology of the present disclosure is the blur correction device according to the first aspect, in which the filter is a filter that removes disturbances that appear in the image due to blurring that occurs during exposure. This makes it possible to more easily remove disturbances that appear in the image due to blurring that occurs during exposure, compared to removing disturbances in the image without using a filter.

[0008] A third aspect of the technique of the present disclosure is the blur correction device according to the first or second aspect, in which the exposure is within one frame. As a result, when blur correction is shared between a mechanical blur correction unit and an electronic blur correction unit, it is possible to remove disturbances that appear in an image for one frame due to blurring that occurs during exposure within one frame without relying on the mechanical blur correction unit.

[0009] A fourth aspect of the technique of the present disclosure is a blur correction device according to any one of the first to third aspects, in which the operation state information is information including at least one of a period during which the optical element is moving during an exposure period and a period during which the optical element is not moving during an exposure period. This makes it possible to improve the accuracy of blur correction by each of the electronic blur correction unit and the supplemental blur correction unit, compared to a case in which neither of the period during which the optical element is moving during an exposure period and the period during which the optical element is not moving during an exposure period is included in the operation state information.

[0010] A fifth aspect of the technique of the present disclosure is the blur correction device according to any one of the first to fourth aspects, in which the operating state information is information including information capable of identifying a movement trajectory of the optical element during an exposure period. This makes it possible to remove disturbances in an image with high accuracy, compared to a case in which disturbances in an image are removed using information unrelated to the movement trajectory of the optical element during an exposure period.

[0011] A sixth aspect of the technique of the present disclosure is a blur correction device according to any one of the first to fifth aspects, in which the image processing includes processing for cutting out an image according to a cut-out position determined according to the operation status information, the ratio, and the amount of blur. This makes it possible to correct blur with high accuracy compared to a case in which processing for cutting out an image is performed according to a cut-out position determined regardless of the operation status information, the preset ratio, and the amount of blur.

[0012] A seventh aspect of the technique of the present disclosure is a blur correction device according to any one of the first to sixth aspects, in which the optical element is at least one of an imaging element and an optical member in which subject light indicating a subject is imaged by the optical member. Thereby, when the optical element is an imaging element, blur can be corrected within the movable range of the imaging element, when the optical element is an optical member, blur can be corrected within the movable range of the optical member, and when the optical element is an imaging element and an optical member, blur can be corrected within the movable range of each of the imaging element and the optical member.

[0013] An eighth aspect of the technique of the present disclosure is a blur correction device according to any one of the first to seventh aspects, in which the detection unit detects the amount of blur for each of a plurality of frequencies of blur, and the ratio is determined according to the frequency. By detecting the amount of blur for each of a plurality of frequencies, it is possible to correct blur with a higher degree of accuracy than when blur is corrected using only the amount of blur for one frequency.

[0014] A ninth aspect of the technique of the present disclosure is a blur correction device according to any one of the first to seventh aspects, in which the detection unit detects the amount of blur in each of a pair of axial directions, and the ratio is determined according to the axial direction. This makes it possible to correct blur with higher accuracy than when blur is corrected using only the amount of blur in one axial direction.

[0015] A tenth aspect of the technique of the present disclosure includes a blur correction device according to any one of the first to ninth aspects, and an imaging unit that generates an image by capturing an image. With this, when blur correction is shared between the mechanical blur correction unit and the electronic blur correction unit, it is possible to remove disturbances that appear in the image due to blurring that occurs during exposure without relying on the mechanical blur correction unit.

[0016] An eleventh aspect of the technique of the present disclosure is a monitoring system including the imaging device according to the tenth aspect, and a control device that performs at least one of control for causing a display unit to display an image reflecting the correction result of the supplemental blur correction unit and control for causing a storage unit to store image data showing the image reflecting the correction result. In this way, when blur correction is shared between the mechanical blur correction unit and the electronic blur correction unit, it is possible to remove disturbances that appear in an image due to blurring that occurs during exposure without relying on the mechanical blur correction unit.

[0017] A twelfth aspect of the technique of the present disclosure is a program for causing a computer to function as a mechanical blur correction unit, an electronic blur correction unit, and a supplemental blur correction unit included in a blur correction device according to any one of the first to ninth aspects. This makes it possible to remove disturbances that appear in an image due to blurring that occurs during exposure without relying on the mechanical blur correction unit when blur correction is shared between the mechanical blur correction unit and the electronic blur correction unit.

[0018] A thirteenth aspect of the technique of the present disclosure is a blur correction device that includes a sensor that detects the amount of blur of an imaging device, and a processor, wherein the processor performs mechanical blur correction processing to correct blur by moving an optical element included in the imaging device based on the amount of blur, performs electronic blur correction processing to correct blur by performing image processing on an image obtained by imaging with the imaging device based on the amount of blur and operating status information regarding the operating status of the optical element during exposure in the imaging device, and performs supplementary blur correction processing to correct blur by applying a filter determined according to the operating status information, ratio, amount of blur, and exposure period in the imaging device when the mechanical blur correction processing and the electronic blur correction processing are operated in a predetermined ratio during exposure. [Brief description of the drawings]

[0019] [Figure 1] 1 is a schematic configuration diagram showing an example of the configuration of a monitoring system according to an embodiment. [Diagram 2] 1 is a block diagram showing an example of the configuration of an optical system and an electrical system of a surveillance camera according to an embodiment. [Diagram 3] FIG. 2 is a block diagram showing an example of a configuration of an electrical system of a management device according to an embodiment. [Figure 4] 2 is a functional block diagram showing an example of functions of a CPU included in the surveillance camera according to the embodiment. FIG. [Diagram 5] 4 is a conceptual diagram showing an example of processing contents of an electronic blur correction unit included in the surveillance camera according to the embodiment. FIG. [Figure 6]10 is a conceptual diagram showing an example of processing contents of a supplementary shake correction unit realized by a CPU included in the surveillance camera according to the embodiment. FIG. [Figure 7] 11 is a conceptual diagram for explaining the processing content in the management device of a supplementary corrected image transmitted to the management device from the surveillance camera according to the embodiment. FIG. [Figure 8] 10 is a flowchart illustrating an example of a flow of a correction control process according to the embodiment. [Figure 9] 9 is a continuation of the flowchart shown in FIG. 8. [Figure 10] FIG. 11 is a functional block diagram showing a modified example of the functions of the CPU included in the surveillance camera according to the embodiment. [Figure 11] 13 is a conceptual diagram showing a modified example of ratio information used in the surveillance camera according to the embodiment. FIG. [Figure 12] 1 is a conceptual diagram showing an example of a manner in which a correction control program according to an embodiment is installed in a computer in a surveillance camera from a storage medium on which the correction control program is stored. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0020] An example of an embodiment of the technology of the present disclosure will be described with reference to the accompanying drawings.

[0021] First, the terms used in the following description will be explained.

[0022] CPU is an abbreviation for "Central Processing Unit." RAM is an abbreviation for "Random Access Memory." ROM is an abbreviation for "Read Only Memory."

[0023] ASIC is an abbreviation for "Application Specific Integrated Circuit". PLD is an abbreviation for "Programmable Logic Device". FPGA is an abbreviation for "Field-Programmable Gate Array". AFE is an abbreviation for "Analog Front End". DSP is an abbreviation for "Digital Signal Processor". SoC is an abbreviation for "System-on-a-chip".

[0024] SSD is an abbreviation for "Solid State Drive." DVD-ROM is an abbreviation for "Digital Versatile Disc Read Only Memory." USB is an abbreviation for "Universal Serial Bus." HDD is an abbreviation for "Hard Disk Drive." EEPROM is an abbreviation for "Electrically Erasable and Programmable Read Only Memory."

[0025] CCD is an abbreviation for "Charge Coupled Device". CMOS is an abbreviation for "Complementary Metal Oxide Semiconductor". EL is an abbreviation for "Electro-Luminescence". A / D is an abbreviation for "Analog / Digital". I / F is an abbreviation for "Interface". UI is an abbreviation for "User Interface". WAN is an abbreviation for "Wide Area Network".

[0026] FIR is an abbreviation for "Finite Impulse Response." IIR is an abbreviation for "Infinite Impulse Reponse." SNR is, for example, an abbreviation for "Signal-to-Noise-Ratio." In the following explanation, when "image" is used in reference to something other than an "image" displayed on a display, "image" also includes the meaning of "data showing an image."

[0027] 1, a surveillance system 2 includes a surveillance camera 10 and a management device 11. The surveillance system 2 is an example of a "surveillance system" according to the technology of the present disclosure, and the surveillance camera 10 is an example of an "imaging device" according to the technology of the present disclosure.

[0028] The surveillance camera 10 is installed on a pillar or a wall indoors or outdoors, captures an image of a subject to be monitored, and generates a moving image by capturing the image. The moving image includes a plurality of frames of images obtained by capturing the image. The surveillance camera 10 transmits the moving image obtained by capturing the image to the management device 11 via a communication line 12.

[0029] The management device 11 includes a display 13 and a secondary storage device 14. Examples of the display 13 include a liquid crystal display and an organic EL display. The display 13 is an example of a "display unit (display)" according to the technology of the present disclosure.

[0030] An example of the secondary storage device 14 is a HDD. The secondary storage device 14 may be a non-volatile memory such as a flash memory, an SSD, or an EEPROM, instead of a HDD. The secondary storage device 14 is an example of a "storage unit (memory)" according to the technology of the present disclosure.

[0031] In the management device 11, the moving images transmitted by the surveillance camera 10 are received, and the received moving images are displayed on the display 13 and / or stored in the secondary storage device .

[0032] As an example, as shown in FIG. 2, the surveillance camera 10 includes an optical system 15 and an image sensor 25. The image sensor 25 is located behind the optical system 15. The optical system 15 includes an objective lens 15A and a lens group 15B. The objective lens 15A and the lens group 15B are arranged in this order along the optical axis OA of the optical system 15 from the monitored object side to the light receiving surface 25A side of the image sensor 25. The lens group 15B includes a focusing lens and a zooming lens that are each movable along the optical axis OA. The focusing lens and the zooming lens move along the optical axis OA according to a given power. The lens group 15B also includes an anti-vibration lens 15B1. The anti-vibration lens 15B1 moves in a direction perpendicular to the optical axis of the anti-vibration lens 15B1 according to a given power. The anti-vibration lens 15B1 is an example of an "optical element" and an "optical member" according to the technology of the present disclosure. In this embodiment, the meaning of "perpendicular" includes not only the meaning of "completely perpendicular" but also the meaning of "approximately perpendicular" that includes tolerances allowed in design and manufacturing.

[0033] The optical system 15 thus configured forms an image of the monitoring target light, which indicates the monitoring target, on the light receiving surface 25A. The imaging element 25 is an example of an "imaging section (image sensor)" according to the technology of the present disclosure. Although a CCD image sensor is used as the imaging element 25 here, this is merely an example, and the imaging element 25 may be another image sensor such as a CMOS image sensor.

[0034] Incidentally, the vibrations given to the surveillance camera 10 include, if outdoors, vibrations due to passing cars, vibrations due to wind, vibrations due to road construction, etc., and if indoors, vibrations due to the operation of an air conditioner, vibrations due to people entering and exiting, etc. Therefore, the surveillance camera 10 becomes blurred due to the vibrations given to the surveillance camera 10 (hereinafter also simply referred to as "vibrations").

[0035] In this embodiment, "blur" refers to a phenomenon in which a digital image fluctuates due to a change in the positional relationship between the optical axis OA and the light receiving surface 25A in the surveillance camera 10. In other words, "blur" can be said to be a phenomenon in which the optical image formed on the light receiving surface 25A fluctuates due to the optical axis OA tilting caused by vibration. The fluctuation of the optical axis OA means, for example, that the optical axis OA tilts with respect to a reference axis (for example, the optical axis OA before the occurrence of blur).

[0036] Therefore, the surveillance camera 10 includes a mechanical shake correction unit 29 to correct the shake. That is, the mechanical shake correction unit 29 is used to correct the shake. The mechanical shake correction unit 29 and an operation control unit 37A (see FIG. 4) described later are an example of a "mechanical shake correction unit (mechanical shake correction device)" according to the technology of the present disclosure. The mechanical shake correction unit is a device that performs mechanical shake correction processing, that is, a device that performs so-called OIS (Optical Image Stabilization). Here, OIS is illustrated, but it may be a device that performs BIS (Body Image Stabilization) instead of OIS, or a device that performs OIS and BIS. In this embodiment, "shake correction" includes the meaning of reducing shake in addition to eliminating shake.

[0037] The mechanical shake correction unit 29 includes an anti-vibration lens 15B1, an actuator 17, a driver 23, and a position detection sensor 39. The anti-vibration lens 15B1 is an example of the "optical element" according to the technique of the present disclosure.

[0038] Various well-known methods can be used to correct blur by the mechanical blur correction unit 29. In this embodiment, the method of correcting blur is to move the vibration-proof lens 15B1 based on the amount of blur detected by a blur amount detection sensor 40 (described later). Specifically, blur correction is performed by moving the vibration-proof lens 15B1 in a direction that cancels the blur, and by an amount that cancels the blur.

[0039] An actuator 17 is attached to the anti-vibration lens 15B1. The actuator 17 is a shift mechanism equipped with a voice coil motor, and by driving the voice coil motor, the anti-vibration lens 15B1 is moved in a direction perpendicular to the optical axis of the anti-vibration lens 15B1. Note that, although a shift mechanism equipped with a voice coil motor is used as the actuator 17 here, the technology of the present disclosure is not limited to this, and other power sources such as a stepping motor or a piezoelectric element may be applied instead of the voice coil motor.

[0040] The actuator 17 is controlled by a driver 23. When the actuator 17 is driven under the control of the driver 23, the position of the vibration-proof lens 15B1 is mechanically moved with respect to the optical axis OA.

[0041] The position detection sensor 39 detects the current position of the anti-vibration lens 15B1 and outputs a position signal indicating the detected current position. Here, a device including a Hall element is used as an example of the position detection sensor 39. Here, the current position of the anti-vibration lens 15B1 refers to the current position in the anti-vibration lens two-dimensional plane. The anti-vibration lens two-dimensional plane refers to a two-dimensional plane perpendicular to the optical axis of the anti-vibration lens 15B1. Note that in this embodiment, a device including a Hall element is used as an example of the position detection sensor 39, but the technology of the present disclosure is not limited to this, and a magnetic sensor, a photo sensor, or the like may be used instead of the Hall element.

[0042] The surveillance camera 10 includes a computer 19, a driver 26, an AFE 30, a DSP 31, an image memory 32, an electronic blur correction unit 33, a communication I / F 34, a blur amount detection sensor 40, a secondary storage device 42, and a UI device 43. The computer 19 includes a RAM 35, a ROM 36, and a CPU 37. The computer 19 is an example of a "computer" according to the technology of the present disclosure. The electronic blur correction unit 33 and an operation control unit 37A (see FIG. 4) described below are an example of an "electronic blur correction unit (electronic blur correction circuit)" according to the technology of the present disclosure. The electronic blur correction unit is a processor that performs electronic blur correction processing, a processor (EIS processor) that performs so-called "EIS (Electric Image Stabilization)".

[0043] The driver 26, the AFE 30, the DSP 31, the image memory 32, the electronic blur correction unit 33, the communication I / F 34, the RAM 35, the ROM 36, the CPU 37, the blur amount detection sensor 40, the secondary storage device 42, and the UI device 43 are connected to a bus line 38. In addition, the driver 23 is also connected to the bus line 38.

[0044] Various programs for the surveillance camera 10 are stored in the ROM 36. The CPU 37 reads out the various programs from the ROM 36, and loads the read out various programs in the RAM 35. The CPU 37 controls the entire surveillance camera 10 in accordance with the various programs loaded in the RAM 35.

[0045] The imaging element 25 is connected to a driver 26 and an AFE 30. The imaging element 25 captures an image of a monitoring target at a default frame rate under the control of the driver 26. The "default frame rate" referred to here indicates, for example, a few frames / second to several tens of frames / second.

[0046] The light receiving surface 25A is formed by a plurality of photosensitive pixels (not shown) arranged in a matrix. In the imaging element 25, each photosensitive pixel is exposed to light, and photoelectric conversion is performed for each photosensitive pixel. The charge obtained by performing photoelectric conversion for each photosensitive pixel is an analog imaging signal indicative of the monitoring target, and is accumulated in the imaging element 25 as an analog image. Each photosensitive pixel is reset by the driver 26 under the control of the CPU 37 at a timing such as before or after the analog image is read out. The exposure period for each photosensitive pixel is determined according to the shutter speed, and the shutter speed is adjusted by controlling the reset timing for each photosensitive pixel and the readout timing of the analog image.

[0047] A vertical synchronization signal and a horizontal synchronization signal are input to the imaging element 25 from the driver 26. The vertical synchronization signal is a signal that specifies the timing to start transmitting one frame of an analog image. The horizontal synchronization signal is a signal that specifies the timing to start outputting one horizontal line of an analog image. The imaging element 25 starts outputting an analog image to the AFE 30 in frame units in accordance with the vertical synchronization signal input from the driver 26, and starts outputting an analog image to the AFE 30 in horizontal line units in accordance with the horizontal synchronization signal input from the driver 26.

[0048] The AFE 30 receives an analog image from the imaging element 25. In other words, the analog image is read out from the imaging element 25 by the AFE 30. The AFE 30 performs analog signal processing such as correlated double sampling and gain adjustment on the analog image, and then performs A / D conversion to generate a digital image, which is a digital imaging signal. That is, in the surveillance camera 10, the imaging element 25 captures an image of the monitoring target, thereby obtaining a digital image showing the monitoring target. Here, the digital image is an example of an "image" according to the technology of the present disclosure.

[0049] In the example shown in FIG. 2, the AFE 30 is provided outside the imaging element 25; however, the technology of the present disclosure is not limited to this. The AFE 30 may be integrated into the imaging element 25.

[0050] The DSP 31 performs various types of digital signal processing on the digital image. The various types of digital signal processing refer to, for example, demosaic processing, noise removal processing, gradation correction processing, color correction processing, etc. The DSP 31 outputs the digital image after the digital signal processing to the image memory 32 for each frame. The image memory 32 stores the digital image from the DSP 31.

[0051] The shake amount detection sensor 40 is, for example, a device including a gyro sensor, and detects the amount of shake of the surveillance camera 10. In other words, the shake amount detection sensor 40 detects the amount of shake in each of a pair of axial directions. The gyro sensor detects the amount of rotational shake around each axis (see FIG. 1) of the pitch axis PA, the yaw axis YA, and the roll axis RA (axis parallel to the optical axis OA). The shake amount detection sensor 40 detects the amount of shake of the surveillance camera 10 by converting the amount of rotational shake around the pitch axis PA and the amount of rotational shake around the yaw axis YA detected by the gyro sensor into the amount of shake in a two-dimensional plane parallel to the pitch axis PA and the yaw axis YA. The shake amount detection sensor 40 is an example of a "detection unit (sensor)" according to the technology of the present disclosure. The direction of the pitch axis PA and the direction of the yaw axis YA are examples of a "pair of axial directions" according to the technology of the present disclosure. In addition to the meaning of completely parallel, the meaning of "parallel" in this embodiment also includes the meaning of approximately parallel including an allowable error in design and manufacturing.

[0052] Here, a gyro sensor is given as an example of the shake amount detection sensor 40, but this is merely an example, and the shake amount detection sensor 40 may be an acceleration sensor. The acceleration sensor detects the amount of shake in a two-dimensional plane parallel to the pitch axis PA and the yaw axis YA. The shake amount detection sensor 40 outputs the detected amount of shake to the CPU 37.

[0053] In addition, although an example in which the amount of blur is detected by a physical sensor called the blur amount detection sensor 40 is given here, the technology of the present disclosure is not limited to this. For example, a motion vector obtained by comparing chronologically successive digital images stored in the image memory 32 may be used as the amount of blur. Also, the amount of blur that is finally used may be derived based on the amount of blur detected by a physical sensor and the motion vector obtained by image processing.

[0054] The CPU 37 acquires the amount of blur detected by the blur amount detection sensor 40, and controls the mechanical blur correction unit 29 and the electronic blur correction unit 33 based on the acquired amount of blur. The amount of blur detected by the blur amount detection sensor 40 is used for blur correction by each of the mechanical blur correction unit 29 and the electronic blur correction unit 33.

[0055] The electronic blur correction unit 33 is a device including an ASIC. The electronic blur correction unit 33 corrects blur by performing image processing on the digital image in the image memory 32 based on the amount of blur detected by the blur amount detection sensor 40.

[0056] Note that, although a device including an ASIC is exemplified as the electronic blur correction unit 33 here, the technology of the present disclosure is not limited thereto, and may be, for example, a device including an FPGA or a PLD. Also, for example, the electronic blur correction unit 33 may be a device including two or more of an ASIC, an FPGA, and a PLD. Also, a computer including a CPU, a ROM, and a RAM may be adopted as the electronic blur correction unit 33. There may be one or more CPUs. Also, the electronic blur correction unit 33 may be realized by a combination of a hardware configuration and a software configuration.

[0057] The communication I / F 34 is, for example, a network interface, and controls the transmission of various information between the management device 11 and the monitoring camera 10 via a network. An example of the network is a WAN such as the Internet or a public communication network. The communication I / F 34 manages communication between the monitoring camera 10 and the management device 11.

[0058] The secondary storage device 42 is a non-volatile memory, and stores various information under the control of the CPU 37. Examples of the secondary storage device 42 include a flash memory, an SSD, an EEPROM, and an HDD.

[0059] The UI device 43 includes a reception device 43A and a display 43B. The reception device 43A is, for example, a hard key or a touch panel, and receives various instructions from a user. The CPU 37 acquires the various instructions accepted by the reception device 43A and operates according to the acquired instructions.

[0060] The display 43B displays various information under the control of the CPU 37. Examples of the various information displayed on the display 43B include the contents of various instructions accepted by the accepting device 43A, digital images, and the like.

[0061] 3, the management device 11 includes a display 13, a secondary storage device 14, a control device 60, a receiving device 62, and a communication I / F 66. The control device 60 includes a CPU 60A, a ROM 60B, and a RAM 60C. Each of the receiving device 62, the display 13, the secondary storage device 14, the CPU 60A, the ROM 60B, the RAM 60C, and the communication I / F 66 is connected to a bus line 70.

[0062] ROM 60B stores various programs for management device 11 (hereinafter simply referred to as "management device programs"). CPU 60A reads out the management device programs from ROM 60B and loads the read management device programs in RAM 60C. CPU 60A controls the entire management device 11 in accordance with the management device programs loaded in RAM 60C.

[0063] The communication I / F 66 is, for example, a network interface. The communication I / F 66 is communicatively connected to the communication I / F 34 of the management device 11 via the network, and controls the transmission of various information between the management device 11. For example, the communication I / F 66 requests the management device 11 to transmit a digital image, and receives the digital image transmitted from the communication I / F 34 of the management device 11 in response to the request to transmit the digital image.

[0064] The receiving device 62 is, for example, a keyboard, a mouse, a touch panel, etc., and receives various instructions from a user. The CPU 60A acquires the various instructions received by the receiving device 62, and operates according to the acquired instructions.

[0065] The display 13 displays various information under the control of the CPU 60A. Examples of the various information displayed on the display 13 include the contents of various instructions accepted by the accepting device 62 and digital images received by the communication I / F 66.

[0066] The secondary storage device 14 is a non-volatile memory, and stores various information under the control of the CPU 60 A. Examples of the various information stored in the secondary storage device 14 include digital images received by the communication I / F 66.

[0067] In this way, the control device 60 controls the display 13 to display the digital image received by the communication I / F 66, and controls the secondary storage device 14 to store the digital image received by the communication I / F 66.

[0068] Note that, here, the digital image is displayed on the display 13, and the digital image received by the communication I / F 66 is stored in the secondary storage device 14, but the technology of the present disclosure is not limited to this. For example, the digital image may be either displayed on the display 13 or stored in the secondary storage device 14.

[0069] As an example, as shown in Fig. 4, a correction control program 36A is stored in the ROM 36. The CPU 37 reads out the correction control program 36A from the ROM 36, and loads the read correction control program 36A in the RAM 35. The CPU 37 executes the correction control program 36A loaded in the RAM 35 to operate as an operation control unit 37A, a supplementary shake correction unit 37B, and a transmission unit 37C. The supplementary shake correction unit 37B is an example of a "supplementary shake correction unit (supplementary shake correction circuit)" according to the technology of the present disclosure. The supplementary shake correction unit 37B is a processor that performs supplementary shake correction processing (supplementary shake correction processing).

[0070] The secondary storage device 42 stores ratio information 42A. The ratio information 42A is information indicating a preset ratio (hereinafter also simply referred to as "ratio") used when the mechanical blur correction unit 29 and the electronic blur correction unit 33 are operated in a shared manner at a preset ratio during exposure of the image sensor 25. "During exposure" here refers to exposure within one frame. Also, the "ratio" here refers to a ratio indicating what percentage of "10" is allocated to the mechanical blur correction unit 29 and the electronic blur correction unit 33, assuming that the degree of correction that can completely cancel the amount of blur detected by the blur amount detection sensor 40 is "10."

[0071] The ratio is determined according to each of the pair of axial directions. That is, the ratio is determined according to each of the pitch axis PA direction and the yaw axis YA direction. In the example shown in FIG. 3, the ratio in the pitch axis PA direction is "mechanical shake correction unit 29: electronic shake correction unit 33 = X1: Y1", and the ratio in the yaw axis YA direction is "mechanical shake correction unit 29: electronic shake correction unit 33 = X2: Y2". When there is a limit to the movable range of the vibration-proof lens 15B1 in the pitch axis PA direction, it is difficult to make the mechanical shake correction unit 29 correct all of the shake amount, so for example, "X1" is set to "6" indicating the degree of correction that can be corrected within the movable range, and "X2" is set to "4". The ratios "X2" and "Y2" in the yaw axis YA direction can also be determined in the same manner as "X1" and "X2".

[0072] The operation control unit 37A obtains ratio information 42A from the secondary storage device 42. The operation control unit 37A also obtains the amount of blur from the blur amount detection sensor 40. Furthermore, the operation control unit 37A obtains a position signal from the position detection sensor 39.

[0073] Operation control unit 37A operates mechanical shake correction unit 29 and electronic shake correction unit 33 in a ratio indicated by ratio information 42A during exposure of image sensor 25. Mechanical shake correction unit 29 moves vibration-proof lens 15B1 within the above-mentioned vibration-proof lens two-dimensional plane based on the amount of shake determined according to the ratio of the amount of shake detected by shake amount detection sensor 40 for each of the pitch axis PA direction and the yaw axis YA direction. Shake is corrected by moving vibration-proof lens 15B1 within the vibration-proof lens two-dimensional plane.

[0074] The operation control unit 37A acquires position signals from the position detection sensor 39 multiple times during exposure of the image sensor 25, and generates operation state information on the operation state of the anti-vibration lens 15B1 during exposure of the image sensor 25 from the acquired multiple position signals. Examples of the operation state information include information that can identify the movement trajectory of the optical axis of the anti-vibration lens 15B1 in the anti-vibration lens two-dimensional plane during exposure of the image sensor 25. Examples of information that can identify the movement trajectory include two-dimensional coordinates that can identify the position in the anti-vibration lens two-dimensional plane. The operation state information is generated for each of the pitch axis PA direction and the yaw axis YA direction. The above-mentioned "movement trajectory" is an example of the "movement trajectory of the optical element during the exposure period" according to the technology of the present disclosure.

[0075] In addition, although the movement locus of the optical axis of the anti-vibration lens 15B1 in a two-dimensional plane of the anti-vibration lens is illustrated here, the technology of the present disclosure is not limited to this. For example, instead of the movement locus of the optical axis of the anti-vibration lens 15B1, the movement locus of the center of the upper end face or the center of the lower end face of the anti-vibration lens 15B1 may be used, as long as it is a movement locus of a location that can be specified as the movement locus of the anti-vibration lens 15B1.

[0076] Electronic blur correction unit 33 acquires one frame's worth of digital image from image memory 32. Electronic blur correction unit 33 performs image processing on one frame's worth of digital image for each of the pitch axis PA direction and the yaw axis YA direction based on operation state information and a blur amount determined according to a ratio of the amount of blur detected by blur amount detection sensor 40. In this way, image processing (described in detail later) is performed on one frame's worth of digital image, thereby correcting blur. The corrected image obtained by correcting blur on one frame's worth of digital image by electronic blur correction unit 33 is output to supplementary blur correction unit 37B.

[0077] Similar to the operation control unit 37A, the supplementary shake correction unit 37B acquires a position signal from the position detection sensor 39 multiple times, and generates operation state information from the acquired multiple position signals. Also, similar to the operation control unit 37A, the supplementary shake correction unit 37B acquires ratio information 42A from the secondary storage device 42. Also, similar to the operation control unit 37A, the supplementary shake correction unit 37B acquires the amount of blur from the blur amount detection sensor 40. Furthermore, the supplementary shake correction unit 37B acquires the exposure period of the image sensor 25 (hereinafter, also simply referred to as the "exposure period"). The exposure period is a period during which exposure is performed on the image sensor 25, and is obtained by measuring the time from the start timing of exposure on the image sensor 25 to the end timing of exposure. Note that, hereinafter, the start timing of exposure on the image sensor 25 is simply referred to as the "exposure start timing", and the end timing of exposure on the image sensor 25 is simply referred to as the "exposure end timing".

[0078] The supplementary blur correction unit 37B acquires a corrected image from the electronic blur correction unit 33. The supplementary blur correction unit 37B corrects blur by applying a filter to the corrected image that is determined according to the operating state information, the ratio indicated by the ratio information 42A, the amount of blur, and the exposure period. The supplementary corrected image obtained by the supplementary blur correction unit 37B correcting blur on the corrected image is output to the transmission unit 37C. The transmission unit 37C transmits the supplementary corrected image to the management device 11.

[0079] In the electronic blur correction unit 33, a process including an image cropping process is adopted as an example of image processing for a digital image. The image cropping process refers to a process of cropping a digital image according to a cropping position determined according to the operation status information, the ratio indicated by the ratio information 42A, and the amount of blur. In other words, the image cropping process is a process of cropping a part of an image area from a digital image stored in the image memory 32 as a corrected image based on the operation status information, the ratio indicated by the ratio information 42A, and the amount of blur.

[0080] When image cropping is performed, as shown in FIG. 5 as an example, imaging area 25B1 of image sensor 25 is set wider than area 25B2 (hereinafter simply referred to as "image output area 25B2") to be output as a blur-corrected digital image, and then imaging is performed. In electronic blur correction unit 33, cropping area specifying information is generated based on the operation state information, the ratio indicated by ratio information 42A, and the amount of blur. The cropping area specifying information refers to information that specifies an image area to be cropped as a corrected image within the digital image. In electronic blur correction unit 33, the image area specified by the cropping area specifying information is cropped as a corrected image from the digital image stored in image memory 32.

[0081] 6, a disturbance N appears in the corrected image obtained by performing image processing on a digital image by electronic blur correction unit 33. The disturbance N occurs due to blurring during exposure of image sensor 25. The image extraction process performed by electronic blur correction unit 33 is a process for extracting a partial image region (a part of the digital image) from the digital image as a corrected image, and does not accommodate blurring that occurs during exposure, so the disturbance N appears in the corrected image due to blurring during exposure.

[0082] Therefore, in this embodiment, for blur correction, supplementary blur correction unit 37B performs supplementary processing to remove disturbance N from the corrected image. That is, supplementary blur correction unit 37B generates a filter according to the operation state information, the ratio indicated by ratio information 42A, the amount of blur, and the exposure period. The "filter" referred to here is a filter that removes disturbance N that appears in the supplementary image due to blurring that occurs during exposure.

[0083] The filter is, for example, an FIR filter. The FIR filter itself is a series of real values ​​including positive and negative values, the number of rows of the series is called the number of taps, and the real values ​​themselves are called tap coefficients. In the supplemental shake correction unit 37B, a tap coefficient determination arithmetic expression is previously determined in which the operation state information, the ratio indicated by the ratio information 42A, the amount of blur, and the exposure period are each an independent variable, and the tap coefficient is a dependent variable, and the tap coefficient is determined using the tap coefficient determination arithmetic expression.

[0084] The tap coefficient determination formula is a formula derived in advance as a formula for removing the disturbance N appearing in the corrected image, for example, based on the results of tests and / or simulations performed using an actual device. Note that "the disturbance N is removed" here means, for example, that a digital image with an infinite signal-to-noise ratio (SNR), that is, a digital image that does not contain any blur including the disturbance N, is obtained.

[0085] In the supplemental blur correction unit 37B, an FIR filter having tap coefficients determined using the tap coefficient determination arithmetic expression is applied to the corrected image, thereby removing the disturbance N from the corrected image. That is, the tap coefficients determined using the tap coefficient determination arithmetic expression are subjected to a convolution operation (product-sum operation) on the corrected image, thereby removing the disturbance N from the corrected image. In this manner, filtering is performed on the corrected image using the FIR filter, thereby generating a supplemental corrected image in which the disturbance N has been removed from the corrected image.

[0086] In the present embodiment, an example in which the tap coefficients are determined using a tap coefficient determination formula is described, but the tap coefficients may be determined using a tap coefficient determination table in which the operation state information, the ratio indicated by the ratio information 42A, the amount of blur, and the exposure period are associated with the tap coefficients. In addition, although an FIR filter is exemplified here, the technology of the present disclosure is not limited to this, and an IIR filter, for example, may be used. In this case, instead of the tap coefficients, parameters used in the IIR filter may be determined from the operation state information, the ratio indicated by the ratio information 42A, the amount of blur, and the exposure period using a formula or a table.

[0087] In the surveillance camera 10, a supplementary corrected image is generated by the supplementary blur correction unit 37B for each digital image obtained by capturing an image of a monitored object by the imaging element 25 at a default frame rate. As shown in FIG. 7 as an example, the supplementary corrected images obtained by reflecting the correction results in the supplementary blur correction unit 37B in the digital image are sequentially input to the transmission unit 37C. The transmission unit 37C transmits each supplementary corrected image input from the supplementary blur correction unit 37B to the management device 11.

[0088] In the management device 11, the supplementary correction images transmitted from the transmission unit 37C are input sequentially to the control device 60. Then, under the control of the control device 60, the display 13 displays the supplementary correction images sequentially input to the control device 60 as live view images, and the secondary storage device 14 stores the supplementary correction images sequentially input to the control device 60 under the control of the control device 60. Here, an example in which both the display of the supplementary correction images and the storage of the supplementary correction images are performed is given, but the technology of the present disclosure is not limited to this, and either the display of the supplementary correction images or the storage of the supplementary correction images may be performed. The control device 60 is an example of a "control device" according to the technology of the present disclosure.

[0089] Next, the operation of the portion of the monitoring system 2 related to the technique of the present disclosure will be described with reference to Fig. 8 and Fig. 9. Note that Fig. 8 and Fig. 9 show an example of the flow of correction control processing executed by the CPU 37.

[0090] In the correction control process shown in FIG. 8, first, in step ST10, the operation control unit 37A starts the operation of the mechanical blur correction unit 29 and the electronic blur correction unit 33 in a predetermined ratio, and then the correction control process proceeds to step ST12.

[0091] By executing the processing of step ST10, mechanical blur correction unit 29 corrects blur by moving anti-vibration lens 15B1 based on the amount of blur detected by blur amount detection sensor 40. Also, electronic blur correction unit 33 corrects blur by performing image processing on one frame of digital image based on the latest operating state information and the amount of blur detected by blur amount detection sensor 40, and generates a corrected image.

[0092] In step ST12, the supplementary blur correction unit 37B judges whether or not the exposure start timing has arrived. If the exposure start timing has not arrived in step ST12, the judgment is negative, and the correction control process proceeds to step ST36 shown in Fig. 9. If the exposure end timing has arrived in step ST12, the judgment is positive, and the correction control process proceeds to step ST14.

[0093] In step ST14, the supplementary blur correction unit 37B starts measuring the exposure period, and then the correction control process proceeds to step ST16.

[0094] In step ST16, the supplementary shake correction unit 37B starts generating operation status information based on the position signal from the position detection sensor 39, and then the correction control process proceeds to step ST18.

[0095] In step ST18, the supplementary shake correction unit 37B acquires the amount of shake from the shake amount detection sensor 40, and then the correction control process proceeds to step ST20.

[0096] In step ST20, the supplementary blur correction unit 37B judges whether the exposure end timing has arrived. If the exposure end timing has not arrived in step ST20, the judgment is negative, and the judgment of step ST20 is performed again. If the exposure end timing has arrived in step ST20, the judgment is positive, and the correction control process proceeds to step ST22.

[0097] In step ST22, the supplementary blur correction unit 37B finishes measuring the exposure period, and then the correction control process proceeds to step ST24.

[0098] In step ST24, the supplementary shake correction unit 37B finishes generating the motion status information based on the position signal from the position detection sensor 39, and then the correction control process proceeds to step ST26.

[0099] In step ST26, the supplemental blur correction section 37B acquires the ratio information 42A from the secondary storage device 42, and then the correction control process proceeds to step ST28.

[0100] In step ST28, the supplementary shake correction unit 37B generates a filter (e.g., the FIR filter described above) based on the operating status information, the ratio indicated by the ratio information 42A, the amount of shake, and the exposure period, and then the correction control process proceeds to step ST30 shown in FIG. 9.

[0101] The operation status information used in this step ST28 is the operation status information obtained in step ST24. Moreover, the ratio information 42A used in this step ST28 is the ratio information 42A acquired in step ST26. Moreover, the blur amount used in this step ST28 is the blur amount acquired in step ST18. Moreover, the exposure period used in this step ST28 is the exposure period measured in step ST22.

[0102] In step ST30 shown in FIG. 9, the supplementary blur correction section 37B acquires the corrected image generated by the electronic blur correction section 33 during the exposure period, and then the correction control process proceeds to step ST32.

[0103] In step ST32, the supplementary shake correction unit 37B corrects the shake by applying the filter generated in step ST28 to the corrected image acquired in step ST30. That is, the supplementary shake correction unit 37B removes the disturbance N in the corrected image by applying the filter generated in step ST28 to the corrected image acquired in step ST30. After the process of step ST32 is executed, the correction control process proceeds to step ST34.

[0104] In step ST34, the transmission unit 37C transmits to the management device 11 the supplementary corrected image obtained by removing the disturbance N in the corrected image by executing the process of step ST32, and then the correction control process proceeds to step ST36.

[0105] In step ST36, the operation control unit 37A determines whether or not a condition for terminating the correction control process (hereinafter referred to as a "correction control process termination condition") is satisfied. An example of the correction control process termination condition is that an instruction to terminate the correction control process is accepted by the acceptance device 43A. In step ST36, if the correction control process termination condition is not satisfied, the determination is negative, and the correction control process proceeds to step ST12 shown in FIG. 8. In step ST36, if the correction control process termination condition is satisfied, the determination is positive, and the correction control process proceeds to step ST38.

[0106] In step ST38, operation control section 37A ends the operation of mechanical shake correction section 29 and electronic shake correction section 33, and then the correction control process ends.

[0107] As described above, in surveillance camera 10, blur is corrected by moving anti-vibration lens 15B1 based on the amount of blur detected by blur amount detection sensor 40. Also, blur is corrected by electronic blur correction unit 33 performing image processing on the digital image based on the operation state information and the amount of blur. The method of blur correction by electronic blur correction unit 33 is a correction method that is realized by cutting out a digital image according to the amount of blur, and is not a correction method that deals with blur that occurs during exposure.

[0108] Therefore, in surveillance camera 10, in order to correct the blurring that occurs during exposure, when mechanical blur correction unit 29 and electronic blur correction unit 33 are operated in a shared manner at a preset ratio during exposure, a filter is generated according to the operating state information, the preset ratio, the amount of blurring, and the exposure period. Then, supplementary blur correction unit 37B applies the filter to the corrected image to correct the blurring. In other words, applying the filter to the corrected image removes the disturbance N in the corrected image.

[0109] Therefore, according to the surveillance camera 10, when blur correction is shared between the mechanical blur correction unit 29 and the electronic blur correction unit 33, the disturbance N that appears in the corrected image due to blurring that occurs during exposure can be removed without relying on the mechanical blur correction unit 29.

[0110] Furthermore, in the surveillance camera 10, a filter that removes disturbance N that appears in the corrected image due to blurring that occurs during exposure is adopted as the filter applied to the corrected image by the electronic blur correction unit 33. Therefore, compared to removing disturbance N without using a filter, disturbance N that appears in the corrected image due to blurring that occurs during exposure can be removed more easily.

[0111] Furthermore, in surveillance camera 10, the operating state information is information about the operating state of anti-vibration lens 15B1 during exposure within one frame. The filter is determined according to the operating state information, ratio, amount of blur, and exposure period when mechanical shake correction unit 29 and electronic shake correction unit 33 are operated in a predetermined ratio during exposure within one frame. Therefore, when blur correction is performed by mechanical shake correction unit 29 and electronic shake correction unit 33 in a shared manner, disturbance N that appears in a corrected image for one frame due to blur occurring during exposure within one frame can be removed without relying on mechanical shake correction unit 29.

[0112] Furthermore, the surveillance camera 10 employs information including information capable of identifying the movement trajectory of the anti-vibration lens 15B1 during the exposure period as the operating state information. Therefore, the disturbance N in the corrected image can be removed with high accuracy compared to the case where the disturbance N in the corrected image is removed using information unrelated to the movement trajectory of the anti-vibration lens 15B1 during the exposure period.

[0113] Furthermore, in the surveillance camera 10, a corrected image is generated by performing an image cutting process that cuts out a digital image according to a cut-out position determined according to the operation status information, a preset ratio, and the amount of blur. Therefore, blur can be corrected with a higher degree of accuracy than when a digital image is cut out according to a cut-out position determined regardless of the operation status information, the preset ratio, and the amount of blur.

[0114] Furthermore, in surveillance camera 10, blurring is corrected by moving anti-vibration lens 15B1. Therefore, blurring can be corrected within the movable range of anti-vibration lens 15B1.

[0115] Furthermore, in the surveillance camera 10, the amount of blurring is detected for each of the pitch axis PA direction and the yaw axis YA, and a predetermined ratio is determined for each of the pitch axis PA direction and the yaw axis YA. Therefore, blurring can be corrected with higher accuracy than when blurring is corrected using only the amount of blurring in one axial direction.

[0116] In the above embodiment, the blur is corrected by moving the anti-vibration lens 15B1, but the technology of the present disclosure is not limited to this. For example, instead of the anti-vibration lens 15B1, the blur may be corrected by moving the image sensor 25 in a plane parallel to the two-dimensional plane of the anti-vibration lens. In this case, the blur can be corrected within the movable range of the image sensor 25. In addition, since the anti-vibration lens 15B1 is not necessary in this case, the optical system 15 can be made smaller.

[0117] Also, blur may be corrected by moving both the anti-vibration lens 15B1 and the image sensor 25. In this case, blur can be corrected within the range of motion of each of the anti-vibration lens 15B1 and the image sensor 25. In this case, blur that could not be completely corrected because the movement of the anti-vibration lens 15B1 is limited by the range of motion of the anti-vibration lens 15B1 can be corrected by moving the image sensor 25.

[0118] In the above embodiment, the supplementary shake correction unit 37B measures the exposure period in real time, but the technology of the present disclosure is not limited to this. For example, the exposure period may be calculated in advance. In this case, as shown in FIG. 10, the CPU 37 has an exposure period calculation unit 37D. When the shutter speed instruction information instructing the shutter speed is received by the reception device 43A, the exposure period calculation unit 37D calculates the exposure period based on the shutter speed instructed by the shutter speed instruction information. In the supplementary shake correction unit 37B, a filter is generated based on the exposure period calculated by the exposure period calculation unit 37D, the operation state information, the default ratio, and the amount of blur.

[0119] In the above embodiment, the amount of shaking is detected in each of the pair of axial directions, and the ratio is determined according to the axial direction, but the technology of the present disclosure is not limited to this. For example, as shown in FIG. 11, the ratio may be determined according to the frequency of shaking.

[0120] In the example shown in FIG. 11, the frequency of the shake is divided into low and high frequencies. The ratio for the low frequency is "mechanical shake correction unit 29: electronic shake correction unit 33 = X1: Y1", and the ratio for the high frequency is "mechanical shake correction unit 29: electronic shake correction unit 33 = X2: Y2". In this case, the amount of shake is detected for each of the low and high frequencies. The amount of shake of the low frequency is extracted from the amount of shake detected by the shake amount detection sensor 40 using a low-pass filter, and the amount of shake of the high frequency is extracted from the amount of shake detected by the shake amount detection sensor 40 using a high-pass filter. Note that the threshold value for dividing the low frequency and the high frequency may be a fixed value or may be a variable value that can be changed according to an instruction received by the receiving device 43A.

[0121] In this way, by detecting the amount of blur for both low and high frequencies, it is possible to correct blur with higher accuracy than when blur is corrected using only the amount of blur for one frequency.

[0122] Although the frequency is divided into low and high frequencies here, the present invention is not limited to this, and even if the frequency is divided into three or more frequencies, the amount of blur can be detected and the ratio determined in a similar manner.

[0123] In the above embodiment, the operation state information includes information that can identify the movement trajectory of the anti-vibration lens 15B1, but the technology of the present disclosure is not limited to this. The operation state information may include at least one of a period during which the anti-vibration lens 15B1 moves during the exposure period and a period during which the anti-vibration lens 15B1 does not move during the exposure period. This can improve the accuracy of blur correction by each of the electronic blur correction unit 33 and the supplemental blur correction unit 37B, compared to a case in which neither of a period during which the anti-vibration lens 15B1 moves during the exposure period and a period during which the anti-vibration lens 15B1 does not move during the exposure period is included in the operation state information.

[0124] In the above embodiment, a filter is generated for each frame, and the generated filter is applied to the corresponding one frame of the correction image. However, the technology of the present disclosure is not limited to this. For example, one filter may be used across multiple frames. Also, the same filter may be used only during a predetermined time period. The "predetermined time period" may be, for example, a time period determined according to an instruction received by the receiving device 43A, or may be a fixed time period.

[0125] Also, a filter may be generated every several frames. In this case, for example, a filter may be generated every frame for a time period when the frequency of occurrence of vibration is relatively high, and a filter may be generated every several frames for a time period when the frequency of occurrence of vibration is relatively low.

[0126] The "relatively high time zone" and the "relatively low time zone" referred to here may be determined based on statistical data obtained by analyzing a plurality of past digital images accumulated by capturing images of a monitored object by the monitoring camera 10. The "relatively high time zone" and the "relatively low time zone" may be fixed, or may be time zones that can be changed according to instructions received by the reception device 43A or the like.

[0127] In the above embodiment, the correction control program 36A is stored in the ROM 36, but the technology of the present disclosure is not limited to this. For example, as shown in FIG. 12, the correction control program 36A may be stored in a storage medium 100. In this case, the correction control program 36A stored in the storage medium 100 is installed in the computer 19, and the CPU 37 executes the above-mentioned correction control process according to the correction control program 36A. The storage medium 100 is a non-transitory storage medium. An example of the storage medium 100 is any portable storage medium such as an SSD or a USB memory.

[0128] In the example shown in FIG. 12, the CPU 37 is a single CPU, but the technology of the present disclosure is not limited to this, and multiple CPUs may be employed.

[0129] Furthermore, the correction control program 36A may be stored in a storage unit of another computer or a server device or the like connected to the computer 19 via a communication network (not shown), and the correction control program 36A may be downloaded to the computer 19 in response to a request from the above-mentioned surveillance camera 10. In this case, the downloaded correction control program 36A is executed by the CPU 37 of the computer 19.

[0130] Furthermore, in the above embodiment, the CPU 37 is exemplified, but the technology of the present disclosure is not limited to this, and multiple CPUs may be used.

[0131] In the above embodiment, the operation control unit 37A, the supplementary shake correction unit 37B, and the transmission unit 37C are described as being realized by a software configuration of the computer 19, but the technology of the present disclosure is not limited to this. For example, the operation control unit 37A, the supplementary shake correction unit 37B, and the transmission unit 37C may be realized by a device including, for example, an ASIC, an FPGA, and / or a PLD. Also, the operation control unit 37A, the supplementary shake correction unit 37B, and the transmission unit 37C may be realized by a combination of a hardware configuration and a software configuration.

[0132] The hardware resources for executing the above-mentioned correction control process may be various processors as shown below. As an example of the processor, as described above, there is a CPU, which is a general-purpose processor that functions as a hardware resource for executing the correction control process by executing software, i.e., a program. In addition, as an example of the processor, there is a dedicated electric circuit, which is a processor having a circuit configuration designed specifically for executing a specific process, such as an FPGA, a PLD, or an ASIC.

[0133] The hardware resource that executes the correction control process may be one of these various processors, or may be a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the correction control process may be a single processor.

[0134] As an example of a single processor, first, as represented by a computer such as a client or server, one processor is configured by a combination of one or more CPUs and software, and this processor functions as a hardware resource that executes the processes of each of the operation control unit 37A, the supplementary shake correction unit 37B, and the transmission unit 37C. Second, as represented by an SoC, there is a form in which a processor is used that realizes the functions of the entire system including multiple hardware resources that execute the correction control process in one IC chip. In this way, the processes of each of the operation control unit 37A, the supplementary shake correction unit 37B, and the transmission unit 37C are realized using one or more of the above-mentioned various processors as hardware resources.

[0135] Furthermore, more specifically, the hardware structure of these various processors can be an electric circuit that combines circuit elements such as semiconductor elements.

[0136] In the above embodiment, the surveillance camera 10 is given as an example of an imaging device according to the technology of the present disclosure, but the technology of the present disclosure is not limited thereto. For example, instead of the surveillance camera 10, the technology of the present disclosure can be applied to various electronic devices such as a portable interchangeable lens camera, a portable fixed lens camera, a personal computer, a smart device, or a wearable terminal device. Even with these electronic devices, the same actions and effects as those of the surveillance camera 10 described in each of the above embodiments can be obtained.

[0137] The above correction control process is merely an example, and it goes without saying that unnecessary steps may be deleted, new steps may be added, or the process order may be changed without departing from the spirit of the present invention.

[0138] The above description and illustrations are detailed descriptions of the parts related to the technology of the present disclosure, and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, function, action, and effect is an example of the configuration, function, action, and effect of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above description and illustrations, within the scope of the gist of the technology of the present disclosure. In addition, in order to avoid confusion and to facilitate understanding of the parts related to the technology of the present disclosure, the above description and illustrations omit explanations of technical common sense that do not require explanation in order to enable the implementation of the technology of the present disclosure.

[0139] In this specification, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. In addition, in this specification, the same idea as "A and / or B" is also applied when three or more things are expressed by connecting them with "and / or."

[0140] All publications, patent applications, and standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or standard was specifically and individually indicated to be incorporated by reference.

Claims

1. A detection unit that detects an amount of blurring of the imaging device; a mechanical blur correction unit that corrects blur by moving an optical element included in the imaging device based on the amount of blur; an electronic blur correction unit that corrects blur on the basis of the amount of blur for an image captured by the imaging device; a supplementary blur correction unit that corrects the blur by applying an FIR filter or an IIR filter to the image obtained by capturing an image while operating the mechanical blur correction unit and the electronic blur correction unit in a shared manner at a predetermined ratio during exposure by the image capture device, the electronic blur correction unit corrects the blur by performing image processing based on the amount of blur and operational status information relating to an operational status of the optical element during exposure in the imaging device; The FIR filter or the IIR filter is determined in accordance with the motion state information, the ratio, the amount of blur, and an exposure period in the imaging device. Image stabilization device.

2. The FIR filter or the IIR filter is generated every time one frame is captured by the imaging device, the FIR filter or the IIR filter is generated every time multiple frames are captured by the imaging device, the same FIR filter or the IIR filter is used over a period during which multiple frames are captured by the imaging device, or the same FIR filter or the IIR filter is used over a predetermined time period. The image blur correction device according to claim 1 .

3. The FIR filter or the IIR filter is determined according to the operation state information, the ratio, the amount of blur, and an exposure period in the imaging device when the mechanical blur correction unit and the electronic blur correction unit are operated during the exposure. The image blur correction device according to claim 2 .

4. The exposure is within one frame. The blur correction device according to claim 1 .

5. The operating state information is information including at least one of a period during which the optical element is moving during an exposure period and a period during which the optical element is not moving during the exposure period. The blur correction device according to claim 1 .

6. The operating state information includes information that can identify a movement trajectory of the optical element during an exposure period. The blur correction device according to claim 1 .

7. The image processing includes a process of cutting out the image in accordance with a cut-out position determined according to the motion status information, the ratio, and the amount of blur. The blur correction device according to any one of claims 1 to 6.

8. The optical element is at least one of an image pickup element in which subject light representing a subject is imaged by an optical member and the optical member. The blur correction device according to claim 1 .

9. the detection unit detects the amount of shaking for each of a plurality of frequencies of the shaking, The ratio is determined according to the frequency. The blur correction device according to claim 1 .

10. The detection unit detects the amount of shake in each of a pair of axial directions, The ratio is determined according to the axial direction. The blur correction device according to claim 1 .

11. A blur correction device according to any one of claims 1 to 10; An imaging unit that captures an image to generate the image; An imaging device comprising:

12. The imaging device according to claim 11 ; a control device that performs at least one of control for causing a display unit to display the image in which the correction result by the supplemental blur correction unit is reflected, and control for causing a storage unit to store image data representing the image in which the correction result is reflected; and A surveillance system including:

13. Computer, A program for causing the mechanical blur correction unit, the electronic blur correction unit, and the supplemental blur correction unit included in the blur correction device according to claim 1 to function as the mechanical blur correction unit, the electronic blur correction unit, and the supplemental blur correction unit.

Images