Imaging device and control method thereof, and image processing device and method

Abstract

To perform anti-vibration control suitable for moving images and a still image.SOLUTION: An imaging apparatus has: imaging means that photoelectrically converts light incident through a photographic optical system to output an image; shake detection means that detects a shake of the imaging apparatus; and driving means that drives to translate the imaging means in a translation direction on a surface perpendicular to an optical axis of the photographic optical system to correct the shake detected by the shake detection means, and drives to rotate the imaging means in a rotation direction around the optical axis. When a plurality of images obtained from the imaging means are composed, the driving means performs the rotation drive in preference to the translation drive.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an imaging apparatus and a control method thereof, and an image processing apparatus and method, and more particularly to an image stabilization technique in an imaging apparatus. [Background technology]

[0002] In recent years, as imaging devices have become more sophisticated, many imaging devices are equipped with a technology (image stabilization) that corrects camera shake by moving the imaging element. In this image stabilization technology, many image stabilization mechanisms have been proposed that allow rotational movement around the optical axis direction in addition to translational movement within a plane perpendicular to the optical axis.

[0003] Another type of image stabilization is electronic image stabilization, which corrects image stabilization by electronically aligning the captured image so that it overlaps with a reference image. Furthermore, there is a technology that aligns and synthesizes multiple images to obtain an image equivalent to a long exposure (hereafter referred to as "image synthesis image stabilization"). Electronic image stabilization does not require image synthesis, and is often used for videos that require real-time performance. On the other hand, image synthesis image stabilization requires synthesis processing and is designed for long exposures, so it is generally used for still images.

[0004] Furthermore, Patent Document 1 discloses a technique for correcting camera shake by coordinating image plane stabilization, which corrects rotational shake by rotating an image sensor, with electronic stabilization. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2012-242563 A Summary of the Invention [Problem to be solved by the invention]

[0006] Generally, in still image stabilization, the image quality of each image is emphasized, while in video stabilization, the time-series stability of the images is emphasized. Therefore, since the requirements for image synthesis stabilization for still images and electronic stabilization for videos are different, if image plane stabilization and electronic stabilization are performed using the same method, the desired image quality may not be obtained for either the still image or the video image.

[0007] The present invention has been made in consideration of the above problems, and has an object to perform image plane vibration reduction control suitable for both moving images and still images.

[0008] A second object of the present invention is to improve the image quality of a composite image when a plurality of images are composited. [Means for solving the problem]

[0009] In order to achieve the above object, an image pickup apparatus according to the present invention comprises: an image pickup means for photoelectrically converting light incident via an image pickup optical system and outputting an image; A shake detection means for detecting a shake of the imaging device; ,before a driving means for translating and rotating the imaging means in a plane perpendicular to an optical axis of the photographing optical system; a generating means for generating a translation amount of the translation drive and a rotation amount of the rotation drive so as to correct the shake detected by the shake detection means; When a plurality of images output from the imaging means are synthesized, When the plurality of images are captured by the imaging means, the generating means The imaging means Before Rotation drive and generating a rotation amount for rotating the image pickup means after the rotation drive. The translation drive Generate the translation amount as much as possible It is characterized by: Effect of the Invention

[0010] According to the present invention, it is possible to perform image plane stabilization control suitable for both moving images and still images, and when multiple images are combined, it is possible to improve the image quality of the combined image. [Brief description of the drawings]

[0011] [Figure 1] 1 is a central cross-sectional view and a block diagram showing a schematic configuration of an imaging system according to a first embodiment of the present invention. [Diagram 2]FIG. 2 is an exploded perspective view of the vibration isolation mechanism according to the first embodiment. [Diagram 3] 4 is a flowchart illustrating processing according to the first embodiment. [Figure 4] 5A and 5B are diagrams illustrating the relationship between rotation and translation of the vibration isolation mechanism in the first embodiment. [Diagram 5] 13A to 13C are diagrams for explaining tilt blur according to a second embodiment. [Figure 6] 10 is a flowchart illustrating processing according to a second embodiment. [Figure 7] 13A and 13B are diagrams for explaining the relationship between the amount of rotational shake and tilt shake and the focal length according to the second embodiment. [Figure 8] FIG. 11 is a block diagram showing a schematic configuration of an imaging system according to a third embodiment. [Figure 9] 13A to 13C are diagrams illustrating the structure of an image sensor according to a third embodiment. [Figure 10] 13A to 13C are diagrams for explaining processes performed by an image deviation calculator and a reliability detector in the third embodiment. [Figure 11] 13A and 13B are a schematic external view and a block diagram showing the configuration of an image processing unit of an imaging device according to a fourth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, the embodiments will be described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe a number of features, not all of these features are essential to the invention, and the features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicated descriptions are omitted.

[0013] <First embodiment> Fig. 1 shows an image pickup apparatus according to an embodiment of the present invention, Fig. 1(a) is a central cross-sectional view of an image pickup system as an image pickup apparatus according to an embodiment of the present invention, and Fig. 1(b) is a block diagram showing a schematic configuration of the image pickup system.

[0014] 1(a), the imaging system of this embodiment mainly comprises a camera body 1 and a lens unit 2 that is detachable from the camera body 1. The camera body 1 and the lens unit 2 are electrically connected via electrical contacts 11. Note that the imaging device of the present invention is not limited to this configuration, and may be an imaging device in which the camera body and the lens unit are integrally configured.

[0015] The lens unit 2 includes a photographing optical system 3 arranged on an optical axis 4 and consisting of multiple lenses, including a focus lens, a zoom lens, an anti-vibration lens, and an aperture, a lens system control circuit 12, and a lens memory 17. The camera body 1 also includes an image sensor 6, a rear display device 9a, an EVF 9b, an anti-vibration mechanism 14, a shake detection unit 15, and a shutter mechanism 16.

[0016] 1(b) is a diagram showing the electrical configuration of the imaging system, in which the lens unit 2 further includes a lens drive mechanism 13 that drives the focus lens, zoom lens, anti-vibration lens, aperture, etc. included in the photographing optical system 3. The camera body 1 further includes a camera system control circuit 5, an image processing unit 7, a memory 8, a display unit 9, and an operation detection unit 10. The display unit includes a rear display device 9a and an EVF 9b.

[0017] In the imaging system having the above configuration, light from a subject incident via the photographing optical system 3 is imaged on the imaging surface of the imaging element 6. The imaging element 6 photoelectrically converts the incident light and outputs an electrical signal (image signal) according to the amount of light. The imaging element 6 can output image signals of images in various formats such as so-called still images and videos, and can output videos in multiple formats by changing the aspect ratio, the resolution of the recorded image, etc. In addition, the imaging system of this embodiment has a mode (dynamic range expansion, noise reduction, etc.) for acquiring and synthesizing a group of images that are continuous in time for still images, and there are cases where a group of images is acquired that is continuous in time, regardless of whether it is a still image or a video.

[0018] The image processing unit 7 includes an A / D converter, a white balance adjustment circuit, a gamma correction circuit, an interpolation calculation circuit, and the like, and processes the image signal output from the imaging element 6 to generate image data for recording.

[0019] The image processing unit 7 is also connected to the camera system control circuit 5. The camera system control circuit 5 can obtain a focus evaluation value and an exposure amount based on the image signal from the image sensor 6 processed by the image processing unit 7, and obtain a focus position and exposure conditions (F-number, shutter speed, etc.) based on these values. The camera system control circuit 5 then issues a command to the lens system control circuit 12 via an electrical contact 11 based on the obtained focus position and exposure conditions, and the lens system control circuit 12 controls the lens drive mechanism 13 based on the command. The shutter mechanism 16 controls the incidence / blocking of the subject image on the image sensor 6 by moving the shutter curtain. Through the above control, the image sensor 6 is exposed to an appropriate amount of light, and the subject image is formed near the image sensor 6.

[0020] The image processing unit 7 in this embodiment also includes a position alignment unit 71 that performs electronic image stabilization by shifting and aligning the positions of multiple images, and an image synthesis unit 72 that synthesizes multiple images. Specific operations of the position alignment unit 71 and the image synthesis unit 72 will be described later. The image processing unit 7 also compresses data such as images, videos, and audio using a predetermined method.

[0021] The memory 8 includes an image storage section, and the camera system control circuit 5 outputs to the recording section of the memory 8. When the user is looking into the EVF 9b, the display unit 9 turns off the rear display device 9a and presents images and information on the EVF 9b under the control of the camera system control circuit 5, and when the user is not looking into the EVF 9b, the display unit 9 presents images and information on the rear display device 9a. The rear display device 9a is a touch panel and is connected to the operation detection unit 10.

[0022] The shake detection unit 15 can detect translational shake in the translation direction on a plane perpendicular to the optical axis 4 and rotational shake of the device in the rotation direction around the optical axis 4, and may use a vibration gyro or an acceleration sensor. Shake may also be detected by comparing images between frames based on the image signal output from the imaging element 6. The vibration isolation mechanism 14 is a mechanism that drives the imaging element 6 in translation within a plane perpendicular to the optical axis 4 and rotates it around the optical axis 4, and the specific structure thereof will be described later with reference to FIG. 2.

[0023] The camera system control circuit 5 generates and outputs timing signals for imaging. In addition, in response to an external operation detected by the operation detection unit 10, the camera system control circuit 5 controls the imaging process, image processing, and recording / playback process. For example, when the operation detection unit 10 detects the pressing of a shutter release button (not shown), the camera system control circuit 5 controls the driving of the imaging element 6, the operation of the image processing unit 7, the operation of the compression process, and the like in response to this. The camera system control circuit 5 also includes an alignment on / off unit 51 and an image synthesis on / off unit 52 for turning on / off the operations of the alignment unit 71 and the image synthesis unit 72 included in the image processing unit 7, respectively, and further includes an anti-shake control unit 53. The anti-shake control unit 53 generates a target value for the drive amount of the imaging element 6 by the anti-shake mechanism 14 from a signal from the shake detection unit 15 based on the on / off of the alignment and the on / off of the image synthesis, as described later, and performs anti-shake drive (image surface anti-shake) control.

[0024] In addition, in a mode that performs normal image stabilization, the camera system control circuit 5 can also perform known optical image stabilization by controlling the image stabilization lens included in the photographing optical system 3 via the lens drive mechanism 13 based on the signal obtained from the image sensor 6.

[0025] Here, a flow of image plane stabilization control in the imaging system having the above configuration will be briefly described.

[0026] In this embodiment, image stabilization control is performed using a shake detection unit 15 that detects shake, an anti-shake mechanism 14 that performs image stabilization, and an anti-shake control system provided in the camera system control circuit 5. Then, during the so-called composition determination operation, in which the operation detection unit 10 detects an operation (SW1) of pressing the shutter release button (not shown) halfway down to start a preliminary shooting operation, image stabilization is performed using the anti-shake mechanism 14 to facilitate composition determination. That is, image stabilization is performed by controlling the anti-shake mechanism 14 based on a signal from the shake detection unit 15. After that, when the operation detection unit 10 detects an operation (SW2) of pressing the shutter release button completely down to start a shooting operation, image stabilization is performed using the anti-shake mechanism 14 to suppress blurring of the subject image acquired by exposure. When a certain time has passed after exposure, the image stabilization operation is stopped.

[0027] Next, the vibration isolation mechanism 14 of this embodiment will be described with reference to FIG. 2. FIG. 2 is an exploded perspective view of the vibration isolation mechanism 14, which is a mechanism that performs shake correction. Note that there is a separate electrical mechanism for control, but this is not included in the diagram shown in FIG. 2. In FIG. 2, the vertical lines are parallel to the optical axis. In FIG. 2, non-moving members (fixed parts) are numbered in the 100s. Moving members (movable parts) are numbered in the 200s. Furthermore, balls that are sandwiched between the fixed part and the movable part are numbered in the 300s.

[0028] 2, 101 denotes an upper yoke, 102a, 102b, and 102c denote screws, 103a, 103b, 103c, 103d, 103e, and 103f denote upper magnets, 104a and 104b denote auxiliary spacers, 105a, 105b, and 105c denote main spacers, 106a, 106b, and 106c denote fixed part rolling plates, 107a, 107b, 107c, 107d, 107e, and 107f denote lower magnets, 108 denotes a lower yoke, 109a, 109b, and 109c denote screws, and 110 denotes a base plate.

[0029] Reference numeral 201 denotes an FPC, 202a, 202b, and 202c denote mounting positions for position detection elements, 203 denotes a movable PCB, 204a, 204b, and 204c denote movable part rolling plates, 205a, 205b, and 205c denote coils, 206 denotes a movable frame, and 301a, 301b, and 301c denote balls.

[0030] The upper yoke 101, the upper magnets 103a, 103b, 103c, 103d, 103e, and 103f, the lower magnets 107a, 107b, 107c, 107d, 107e, and 107f, and the lower yoke 108 form a magnetic circuit, which is a so-called closed magnetic circuit. The upper magnets 103a, 103b, 103c, 103d, 103e, and 103f are adhered and fixed in a state of being attracted to the upper yoke 101. Similarly, the lower magnets 107a, 107b, 107c, 107d, 107e, and 107f are adhered and fixed in a state of being attracted to the lower yoke 108. The upper magnets 103a, 103b, 103c, 103d, 103e, and 103f and the lower magnets 107a, 107b, 107c, 107d, 107e, and 107f are magnetized in the optical axis direction (up and down direction in FIG. 2). Furthermore, adjacent magnets (magnets 103a and 103b in the positional relationship) are magnetized in different directions. Furthermore, opposing magnets (magnets 103a and 107a in the positional relationship) are magnetized in the same direction. In this way, a strong magnetic flux density is generated between the upper yoke 101 and the lower yoke 108 in the optical axis direction.

[0031] Since a strong attractive force is generated between the upper yoke 101 and the lower yoke 108, the main spacers 105a, 105b, and 105c and the auxiliary spacers 104a and 104b are configured to maintain an appropriate distance. The appropriate distance here is a distance that allows the coils 205a, 205b, and 205c and the FPC 201 to be arranged between the upper magnets 103a, 103b, 103c, 103d, 103e, and 103f and the lower magnets 107a, 107b, 107c, 107d, 107e, and 107f while ensuring an appropriate gap. The main spacers 105a, 105b, and 105c are provided with screw holes, and the upper yoke 101 is fixed to the main spacers 105a, 105b, and 105c by the screws 102a, 102b, and 102c.

[0032] Rubber is provided on the body of the main spacers 105a, 105b, and 105c, forming the mechanical ends (so-called stoppers) of the movable parts.

[0033] Holes are provided in base plate 110 to avoid lower magnets 107a, 107b, 107c, 107d, 107e, and 107f, and the faces of the magnets are configured to protrude from the holes. That is, base plate 110 and lower yoke 108 are fixed with screws 109a, 109b, and 109c, and lower magnets 107a, 107b, 107c, 107d, 107e, and 107f, which have a larger dimension in the thickness direction than base plate 110, are fixed so as to protrude from base plate 110.

[0034] The movable PCB 203 is made of magnesium die-cast or aluminum die-cast, and is lightweight and highly rigid. Each element of the movable part is fixed to the movable PCB 203 to form the movable part. Position detection elements are attached to the FPC 201 at positions indicated by position detection element attachment positions 202a, 202b, and 202c on the surface that is not visible in FIG. 2. For example, a Hall element or the like can be used to detect the position using the magnetic circuit described above. Since the Hall element is small, it is arranged so as to be nested inside the windings of the coils 205a, 205b, and 205c.

[0035] The imaging element 6, coils 205a, 205b, 205c, and Hall elements (not shown) are connected to the movable PCB 203. Electrical communication with the outside is performed via a connector on the movable PCB 203.

[0036] Fixed part rolling plates 106a, 106b, and 106c are bonded to base plate 110, and movable part rolling plates 204a, 204b, and 204c are bonded to movable PCB 203, forming the rolling surfaces of balls 301a, 301b, and 301c. By providing the rolling plates separately, it becomes easy to design the surface roughness, hardness, and other properties to a desired state.

[0037] In the above-mentioned configuration, by passing a current through the coils 205a, 205b, and 205c, a force according to Fleming's left-hand rule is generated, and the movable frame 206 can be moved. In addition, feedback control can be performed by using the signal from the Hall element, which is the position detection element described above. By appropriately controlling the value of the Hall element signal, it is possible to perform translational motion within a plane perpendicular to the optical axis and rotation around the optical axis.

[0038] By driving the Hall element signals at the position detector attachment positions 202b and 202c in opposite phases while keeping the signal of the Hall element at the position detector attachment position 202a constant, it is possible to produce rotational motion approximately around the optical axis 4.

[0039] The magnetic flux density in the optical axis direction is detected at the position detection element attachment positions 202a, 202b, and 202c. The characteristics of the magnetic circuit consisting of the upper magnets 103a, 103b, 103c, 103d, 103e, and 103f and the lower magnets 107a, 107b, 107c, 107d, 107e, and 107f are generally nonlinear. Therefore, the magnetic flux density detected at the position detection element attachment positions 202a, 202b, and 202c does not necessarily have a constant resolution over the entire driving range, and the detection resolution changes. This is because there are positions where the magnetic flux density changes steeply and positions where it changes gently, and the steeper the position, the higher the detection resolution is, that is, the larger the change in magnetic flux density relative to the amount of movement. In the magnetic circuit shown in FIG. 2, the change in magnetic flux density is the largest at the boundary position of the magnets (for example, the boundary position between the upper magnets 103a and 103b), and the detection resolution is high.

[0040] Incidentally, since many proposals have been made regarding the control method for the vibration isolation mechanism 14, a detailed explanation will be omitted here.

[0041] Next, the processing in the camera body 1 having the above configuration will be described with reference to the flowchart shown in FIG.

[0042] In S301, the image synthesis on / off unit 52 in the camera system control circuit 5 controls whether or not to perform image synthesis. Note that the user may be able to directly instruct whether or not to perform image synthesis, or it may be automatically switched depending on the camera settings such as the shooting mode and the shooting conditions such as the shutter speed. For example, image synthesis is performed when the shutter speed is slower than the threshold in the still image shooting mode, and image synthesis is not performed when the shutter speed is faster than the threshold in the still image shooting mode or in the video shooting mode. Note that image synthesis is performed after positioning is performed by the positioning unit 71, so when image synthesis is performed, the positioning on / off unit 51 is also turned on. If image synthesis is to be performed, the process proceeds to S302, and if image synthesis is not to be performed, the process proceeds to S303.

[0043] In S302, the image stabilization control unit 53 of the camera system control circuit 5 controls image stabilization so as to give priority to correcting rotational shake out of translational shake and rotational shake (roll shake). Hereinafter, this control will be referred to as "rotational shake correction priority control." The reason why rotational shake is given priority is to suppress deterioration in image quality that occurs in image registration in S304 and image synthesis in S305, which will be described later. This control is suitable for performing image synthesis stabilization accompanied by image registration in still image shooting, where image quality is important.

[0044] Here, the relationship between rotation and translation of the vibration isolation mechanism 14 in this embodiment will be described with reference to Fig. 4. Fig. 4 is a diagram of the vibration isolation mechanism 14 shown in Fig. 3 as viewed from the direction of the optical axis 4 (the Z-axis direction of the coordinate system in the lower right). The coordinate system in the lower right of Fig. 4 is common to Figs. 4(a), (b), and (c). Fig. 4(a) shows a state in which the movable frame 206 is not moving, Fig. 4(b) shows a state in which the movable frame 206 has moved in the positive X direction without rotation, and Fig. 4(c) shows a state in which the movable frame 206 has moved in the positive X direction with rotation.

[0045] A dashed line 400 drawn in common in Figures 4(a), (b), and (c) indicates a reference line indicating the reference of the fixed part in the X direction, and a dashed line 401 indicates a reference line indicating the reference of the fixed part in the Y direction. In Figure 4(b), a dashed line 402 indicates the reference line after the movable frame 206 has moved in the positive X direction. In Figure 4(c), a dashed line 403 indicates the reference line after the movable frame 206 has rotated, and a dashed line 404 indicates the reference line after the movable frame 206 has moved in the positive X direction. In Figure 4(a), the reference line of the movable part 206 overlaps with the reference line of the fixed part. In other words, it is in a state where it has not moved.

[0046] 4(b) shows a state in which the movable frame 206 is moved in the positive X direction until it contacts the main spacer 105a without rotating the movable frame 206. In other words, the movable frame 206 can move in the positive X direction by an amount equivalent to the distance between the reference line 400 and the reference line 402.

[0047] 4(c) shows a state in which the movable frame 206 is rotated and moved in the positive X direction until it comes into contact with the main spacer 105a. The amount of rotation is indicated by the angle between reference line 401 and reference line 403. The amount of movement of the movable frame 206 in the X direction at this time corresponds to the distance between reference line 400 and reference line 404.

[0048] As is clear from a comparison of Figures 4(b) and 4(c), when rotation occurs, the amount of translation possible in the X direction changes. In order to suppress this change, the mechanical end of the movable part 206 can be provided on the optical axis 4, but this is not easy due to the presence of the image sensor 6 and the processing board behind it. For this reason, the mechanical end of the movable part (main spacers 105a, 105b, and 105c in the examples of Figures 2 and 4) is generally provided at a position different from the optical axis 4.

[0049] 4(c), the gap between the main spacer 105c and the movable frame 206 in the Y direction changes compared to when there is no rotation. In other words, it can be seen that the amount of movement in the positive Y direction also decreases due to rotation.

[0050] Although Fig. 4(b) and (c) do not show the specific relationship between the amount of rotation and the amount of translation, it is clear from Fig. 4(c) that, in general, as the amount of rotation increases, the amount of translation that can be moved decreases. In addition, to obtain a specific relationship, it is sufficient to determine the position of the mechanical end of the movable part and perform a numerical calculation. In order to have a margin for the amount of rotation and the amount of translation, it is sufficient to design the main spacers 105a, 105b, and 105c away from the movable frame 206, but this leads to an increase in the size of the device. Since the camera body 1 is used by a user while being carried around, it is very important not to increase the size of the device.

[0051] As described above, since the rotation amount and the translation amount are in a contradictory relationship, in S302, the upper limit value of the rotational shake compensation amount (rotational drive amount) is set higher than the upper limit value of the translational shake compensation amount (translation drive amount), so that rotational shake is corrected with priority.

[0052] In S304, the alignment unit 71 of the image processing unit 7 detects the positional deviation between the multiple images input from the imaging element 6, and performs geometric deformation on the images so as to correct the positional deviation.

[0053] The positional deviation between images can be detected by calculating the similarity between the images using a known template matching method or the like. In the geometric transformation, the image is subjected to an affine transformation or a projective transformation to correct the positional deviation in the translational or rotational direction of the image. This alignment realizes electronic image stabilization.

[0054] In addition, in a geometric transformation, it is necessary to associate each pixel of an image before and after the transformation. When an image is translated or rotated in sub-pixel units, each pixel does not correspond one-to-one, so the correspondence is generally achieved by using pixels interpolated from multiple surrounding pixels. However, when this pixel interpolation is performed, the image resolution is lost, causing degradation of image quality. Therefore, in order to prevent degradation of image quality, it is desirable to avoid translational movement and rotational movement in sub-pixel units as much as possible. Regarding translational movement, by limiting movement in integer pixel units, it is possible to avoid degradation of image quality due to pixel interpolation, although the alignment accuracy decreases. Furthermore, by using the rotational shake correction priority control of S302, rotational shake is sufficiently corrected by the vibration isolation mechanism 14, making rotational movement unnecessary and preventing degradation of image quality.

[0055] In S305, the image synthesis unit 72 of the image processing unit 7 synthesizes the multiple images aligned in S304 by adding them together. This realizes image synthesis and image stabilization. Note that when each of the multiple images is shot with the correct exposure, it is necessary to take the average after adding the images together.

[0056] On the other hand, when image synthesis is not performed, in S303, the image stabilization control unit 53 controls image stabilization so as to correct translational shake and rotational shake in a well-balanced manner. Hereinafter, this control will be referred to as "standard shake correction control." This control is suitable for image stabilization in video shooting, where time-series stability is important. It is also suitable for still image shooting, when image alignment is not performed, i.e., when image synthesis stabilization is not performed.

[0057] In standard shake compensation control, the upper limit of the amount of rotational shake compensation is set lower than in the rotational shake compensation priority control in S302, and is approximately the same as the upper limit of the amount of translational shake compensation, thereby achieving a good balance between compensation of translational shake and rotational shake.

[0058] Next, in S306, the alignment on / off unit 51 of the camera system control circuit 5 controls whether or not to perform image alignment. Whether or not to perform image alignment may be set directly by the user, or may be automatically switched depending on the camera settings and shooting conditions. For example, image alignment is performed in video shooting mode, and image alignment is not performed when the shutter speed is faster than a threshold in still image shooting mode.

[0059] If image alignment, i.e., electronic image stabilization is to be performed, the process proceeds to S307, and if electronic image stabilization is not to be performed, the process ends.

[0060] In S307, the same image registration process as in S304 is performed, and the process ends.

[0061] As described above, according to the first embodiment, when image synthesis image stabilization is performed, image plane image stabilization is set to rotational blur correction priority control, thereby enabling image stabilization suitable for still images that prioritize image quality. Also, when image synthesis image stabilization is not performed, image plane image stabilization is set to standard blur correction control, thereby enabling image stabilization suitable for moving images that prioritize stability.

[0062] In the above embodiment, an example has been described in which the rotational shake is preferentially corrected by setting the upper limit of the rotational shake compensation amount (rotational drive amount) higher than the upper limit of the translational shake compensation amount (translational drive amount) in the rotational shake compensation priority control. However, the method of correcting the rotational shake with priority over the translational shake is not limited to this. For example, the rotational shake compensation amount may be calculated first, and the translational shake may be corrected within the remaining movable range. In addition, when the total of the rotational shake compensation amount and the translational shake compensation amount exceeds the drivable range, the gain used when acquiring the rotational shake compensation amount based on the detected rotational shake may be made larger than the gain used when acquiring the translational shake compensation amount based on the detected translational shake, so that the rotational shake is preferentially corrected. The gain may be set to a correction amount such that the detected shake is 100% corrected when the gain is 1.

[0063] <Second embodiment> Next, a second embodiment of the present invention will be described. In the first embodiment, the vibration isolation mechanism 14 is a mechanism that translates the image sensor 6 within a plane perpendicular to the optical axis 4 and rotates it about the optical axis 4. In the second embodiment, in addition to the configuration shown in FIG. 4, the vibration isolation mechanism 14 further has a mechanism that rotates it about two axes perpendicular to the optical axis 4. This mechanism makes it possible to correct not only translational shake and rotational shake in image plane vibration isolation, but also tilt components caused by shake in the yaw and pitch directions (hereinafter referred to as "tilt shake"). Note that the overall configuration of the imaging system is the same as that in FIG. 1, so a description thereof will be omitted.

[0064] Fig. 5 is a diagram for explaining tilt shake. Fig. 5(a) shows a state when shaking occurs in the yaw direction. 501 indicates the original image, and 502 indicates the image after shaking in the yaw direction. When shaking occurs in the camera in the yaw direction like this, tilt shake α occurs in addition to translation shake Tx in the image, causing it to be distorted into a trapezoid shape.

[0065] Similarly, Fig. 5(b) shows the state when shaking in the pitch direction occurs. 501 shows the original image, and 503 shows the image after shaking in the yaw direction. When shaking in the pitch direction occurs in this way, the image is distorted into a trapezoid shape due to tilt shake β in addition to translation shake Ty.

[0066] Next, the image plane stabilization operation according to the second embodiment will be described with reference to the flowchart shown in Fig. 6. Note that in Fig. 6, the same processes as those shown in Fig. 3 are given the same step numbers, and descriptions thereof will be omitted as appropriate.

[0067] In the second embodiment, when image synthesis is performed, in S601, the image stabilization control unit 53 of the camera system control circuit 5 controls image stabilization so as to give priority to correcting a specific type of shake among translational shake, rotational shake, and tilt shake. Hereinafter, this control will be referred to as "specific shake correction priority control."

[0068] When attempting to correct tilt blur during image alignment, pixel interpolation occurs, just as with rotational blur, resulting in degradation of image quality. For this reason, when performing image synthesis image stabilization, it is preferable to correct tilt blur in addition to rotational blur using image plane stabilization.

[0069] The priority of correction for rotational shake and tilt shake differs depending on the focal length. Figure 7 shows the relationship between the amount of rotational shake and tilt shake and the focal length. In areas where the focal length is shorter than ft, tilt shake becomes dominant, and in areas where the focal length is longer than ft, rotational shake becomes dominant. Therefore, when the focal length is shorter than a predetermined focal length, correction is prioritized in the order of tilt shake > rotational shake > translational shake, and when the focal length is equal to or greater than the predetermined focal length, correction is prioritized in the order of rotational shake > tilt shake > translational shake.

[0070] As described above, according to the second embodiment, by correcting tilt shake in addition to rotation shake by image plane vibration reduction, it is possible to perform image synthesis vibration reduction with less degradation in image quality.

[0071] <Third embodiment> Next, a third embodiment of the present invention will be described. When correcting camera shake by synthesizing multiple images as in the first and second embodiments, if the images are not aligned properly before synthesis and are misaligned, a double image will be generated. Therefore, in this embodiment, a case where the reliability of alignment is determined will be described.

[0072] Fig. 8 is a diagram showing a schematic configuration of an imaging system in the third embodiment. Compared to the configuration of the imaging system shown in Fig. 1(b), it is different in that an image processing unit 7' further includes an image shift calculator 74 and a reliability detector 75. The rest of the configuration is the same as that described with reference to Figs. 1 and 2, so the description will be omitted.

[0073] Fig. 9 is a diagram showing the configuration of an image sensor 6 in the third embodiment. Fig. 9(a) is an enlarged view of some pixels 20 (4x4 pixels) of the image sensor 6 as viewed from the optical axis 4 direction, and Fig. 9(b) is a diagram showing a cross-sectional view (lower part) of one pixel 20 and the correspondence with the exit pupil of the photographing lens. Note that the projection directions are different between the lower and upper parts of Fig. 9(b), but this illustration method is used to make it easier to understand the correspondence state on the pupil plane.

[0074] In the example shown in Fig. 9(a), the image sensor 6 is covered with a color filter in a so-called Bayer array, and R, G, and B indicate the colors of the color filters, red, green, and blue, respectively. The circle in each pixel 20 indicates a microlens 25, and the vertical line in each pixel 20 indicates that the light receiving section of each pixel 20 is divided into two light receiving sections 22a and 22b (photoelectric conversion sections) in the X direction. The arrow 21 indicates the distance between pixels covered with filters of the same color.

[0075] 9B, reference numerals 23a and 23b denote areas on the pupil plane of the photographing optical system 3 corresponding to the light receiving sections 22a and 22b, respectively, and reference numeral 24 denotes an exit pupil of the photographing optical system 3. In FIG.

[0076] As shown in FIG. 9(b), the light receiving sections 22a, 22b of the pixel 20 are conjugated to respective regions 23a, 23b of the pupil plane of the photographing optical system 3 by the microlens 25, and are configured to receive light beams that have passed through different pupil regions of the exit pupil 24.

[0077] From the imaging element 6 having the above configuration, signals having parallax (or signals from different viewpoints) obtained by the light receiving sections 22a and 22b are acquired, respectively, and a phase difference calculation is performed, thereby enabling a known phase difference AF to be performed. The phase difference AF calculation is performed by the camera system control circuit 5. Also, by adding and outputting the signals obtained by the light receiving sections 22a and 22b for each pixel 20, a normal pixel signal can be obtained. In addition, in FIG. 9(b), the areas 23a and 23b are shown with a gap therebetween to clearly indicate that they are different areas, but in reality, they may be in contact with each other or may have overlapping areas. Hereinafter, an image obtained by collecting pixel signals obtained from the light receiving sections 22a of each pixel 20 will be called an "A image," and an image obtained by collecting pixel signals obtained from the light receiving sections 22b of each pixel 20 will be called an "B image."

[0078] Next, the processes performed by the image deviation calculator 74 and the reliability detector 75 in the third embodiment will be described with reference to FIG. The horizontal axis of FIG. 10(a) indicates time. The time when acquisition started is designated as t1, and thereafter, images A and B are acquired successively at t2, t3, and so on, at a predetermined cycle. The image A acquired at time t is designated as A1, and the image B acquired at time t is designated as B1. Similarly, the image A acquired at time t is designated as A1, and the image B acquired at time t is designated as B1. n The image A acquired at n , B image B n Let us assume that.

[0079] Figures 10(b) and (c) show an example in which a circulation path is formed between the images A and B acquired after time t2 and the images A and B acquired previously. Figure 10(d), as a comparison with Figures 10(b) and (c), shows an example in which a circulation path is not formed between the images A and B acquired after time t2 and the images A and B acquired previously.

[0080] In Fig. 10(b) to (d), at time t n-1 and n Consider the images A and B acquired at time t n-1 The images A and B acquired at time t are used as templates for calculating the motion vector. n-1If the image is rejected as a template, then at time t n-1 In some cases, the template may be a previous image. Template rejection is discussed below.

[0081] In Fig. 10(b), at time t n At this timing, A n From B n Movement vector 30, B n From B n-1 Movement vector 31, A n-1 From A n As can be seen from the figure, by repeating the calculation of the above movement vector, n-1 From A n-1 The movement vector 32 moving towards n-1 Since the movement vector in the opposite direction has been calculated at the time of, it can be used by inverting the sign. The important thing here is that the four movement vectors 30, 31, 32, and 33 are n without following the same route. n In the example of FIG. 10(b), it is clear that the image group (A n , B n ), the image group acquired before that (A n-1 , B n-1 A circulation path is formed between the

[0082] Similarly, in the example of FIG. 10(c), at time t n At this timing, A n From B n Movement vector 30, B n From A n-1 Movement vector 34, B n-1 From A n The motion vector 56 moving toward the time t n-1As can be seen from the figure, these four movement vectors 30, 34, 35, and 36 form a circulating path. The image deviation detector 54 of this embodiment operates to form a circulating path in this manner.

[0083] As shown in the examples of Figures 10(b) and (c), there are multiple possible ways to form a circulation path, and it is best to set it as convenient. However, comparisons between images A and B, or between images captured at the same time, are convenient because they are stable against vignetting and subject conditions. In other words, the example shown in Figure 10(b) is a more preferable example.

[0084] FIG. 10D shows the operation of detecting an image shift that is normally performed. Unlike the present embodiment, in the example of FIG. 10D, n At the timing of A n and B n The image obtained by adding (A n +B n ) is generated. Then, the template image A n-1 +B n-1 As is clear from FIG. 10(d), no route that allows circulation is formed.

[0085] Next, a method for using the circulating path will be described. In this embodiment, the reliability detector 75 circulates and accumulates the movement vector to the original image again. If there is no detection error, the accumulation result will be zero since the image will return to its original position. On the other hand, if the detection fails for some reason, such as the movement of the subject, a mistake in a repeating pattern, or a change in the ambient light, the result will not be zero.

[0086] Therefore, if the result of the reliability detector 75 is greater than the threshold, it is determined that a detection error has occurred. n At time t n If you want to align and combine images, you can decide whether to add them or not based on the quality judgment at this point, which saves memory.

[0087] In addition, based on the comparison with the threshold, it is possible to determine whether or not the template should be updated. n The image of the frame is used as a template for the next image to be acquired, and if it is greater than the threshold, the template is not updated. This makes it possible to perform stable synthesis processing by excluding a single frame with different characteristics (e.g., positioning fails due to flickering).

[0088] In the above example, the light receiving portion of each pixel 20 is divided into two light receiving portions 22a and 22b in the X direction, but the present invention is not limited to this. For example, it may be divided in the Y direction, or the division directions may be mixed. The number of divisions is not limited to two, and it may be divided into three or more. In either case, the reliability of the alignment can be determined by using the method of using the movement vector between the images as described above.

[0089] In this embodiment, the imaging device is provided with the image shift calculator 74 and the reliability detector 75, but the present invention is not limited to this. As another configuration, for example, the image shift calculator 74 and the reliability detector 75 may be used to store the image group A and the image group B, and process them later, using an optical system as shown in this embodiment. In this case, the image shift calculator 74 and the reliability detector 75 are realized in the form of a computer program or the like, and process the stored image group A and the image group B. The processing content is as shown in this embodiment, where the image shift amount is calculated for the image group after time t2 so as to form a circular path between the image group before that time and the image group before that time, and the detection amount along the above-mentioned circular path is integrated. By using this result, n At time t n In this case, as in the example of the imaging device, memory can be saved and memory access can be reduced to perform calculations at high speed.

[0090] As described above, according to the third embodiment, the reliability of alignment can be determined with a simple configuration.

[0091] <Fourth embodiment> Next, a fourth embodiment of the present invention will be described. FIG. 11(a) is a schematic external view showing a smartphone 41 as an imaging device in the fourth embodiment, and FIG. 11(b) is a block diagram mainly showing the configuration of an image processing unit 7". As the smartphone 41 in the fourth embodiment, a so-called multi-lens camera having two shooting optical systems 3a, 3b and two image sensors 6a, 6b will be described. The shooting optical systems 3a, 3b have different focal lengths and can capture images with different angles of view. Here, it is assumed that the shooting optical system 3b has a shorter focal length. In addition, since the shooting optical systems 3a and 3b are separated from each other, images can be captured from multiple viewpoints simultaneously. 4a and 4b indicate the optical axes of the shooting optical systems 3a and 3b, respectively.

[0092] The circuit configuration other than that described above is similar to that shown in FIG. 1(b). However, the processing in the image processing unit 7" is different, which will be described below. The image processing unit 7" in this embodiment includes a magnification adjuster 73 in addition to the configuration of the image processing unit 7' in FIG. 8.

[0093] In this embodiment, the image obtained from the imaging element 6a is referred to as image A. Meanwhile, the image obtained from the imaging element 6b is sent to a magnification adjuster 73 and converted into a state that can be compared with the imaging element 6a. Specifically, a part of the image obtained at a wider angle is cut out, and a process is performed to match the pixel pitch. The output image of this magnification adjuster 73 is referred to as image B.

[0094] Thereafter, the acquired images A and B are used to process the images A and B in the manner described in the third embodiment with reference to Fig. 10. That is, the image shift calculator 74 finds the image shift amount for the second and subsequent image groups so as to form a cyclic path between the image groups before that. Then, the reliability detector 75 accumulates the detection amount along the cyclic path. As a result, the same effect as in the third embodiment can be obtained.

[0095] In this embodiment, the smartphone 41 has been described as having two shooting optical systems 3a, 3b and two image sensors 6a, 6b. However, the imaging system may be three or more systems, in which case, it is sufficient to provide the required number of magnification adjusters 73.

[0096] Furthermore, the present embodiment is not limited to being applied to smartphones, but can also be applied to electronic devices capable of processing images.

[0097] Furthermore, the present invention may be applied to a system made up of a plurality of devices, or to an apparatus made up of a single device.

[0098] <Other embodiments> The present invention can also be realized by supplying a program for implementing one or more of the functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0099] The invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0100] 1: camera body, 2: lens unit, 3: photographing optical system, 4: optical axis, 5: camera system control circuit, 6: image sensor, 7: image processing unit, 10: operation detection unit, 14: vibration isolation mechanism, 15: blur detection unit, 51: on / off unit, 52: image synthesis on / off unit, 53: vibration isolation control unit, 71: position alignment unit, 72: image synthesis unit, 73: magnification adjuster, 74: image deviation calculator, 75: reliability detector

Claims

1. an imaging means for photoelectrically converting light incident via an imaging optical system to output an image; A shake detection means for detecting a shake of the imaging device; a driving means for translating and rotating the imaging means in a plane perpendicular to an optical axis of the photographing optical system; a generating unit that generates a translation amount of the translation drive and a rotation amount of the rotation drive so as to correct the shake detected by the shake detection unit, an imaging device characterized in that, when a plurality of images output from the imaging means are combined, when the plurality of images are captured by the imaging means, the generation means generates a rotation amount for driving the imaging means in a rotational manner, and generates the translation amount within a range in which the imaging means can be driven in a translational manner after the rotational drive.

2. 2. The imaging apparatus according to claim 1, wherein the driving means sets the upper limit value of the rotation drive higher when the plurality of images are combined than when the combination is not performed.

3. an imaging means for photoelectrically converting light incident via an imaging optical system to output an image; A shake detection means for detecting a shake of the imaging device; a driving means for translating and rotating the imaging means in a plane perpendicular to an optical axis of the photographing optical system; a generating unit that generates a translation amount of the translation drive and a rotation amount of the rotation drive so as to correct the shake detected by the shake detection unit, an imaging device characterized in that, when multiple images output from the imaging means are combined, the generation means sets an upper limit value of the rotational drive higher and an upper limit value of the translational drive lower when capturing the multiple images by the imaging means than when the combination is not performed.

4. A setting means for setting whether or not the plurality of images are to be synthesized, 4. The imaging device according to claim 1, wherein the setting means sets the multiple images to be combined when a user instructs the image to be combined or when a shutter speed in a still image shooting mode is slower than a threshold value.

5. A setting means for setting whether or not the plurality of images are to be synthesized, The imaging device according to any one of claims 1 to 4, characterized in that the setting means sets the image to not be combined when the user does not instruct image combination, when the shutter speed is faster than a threshold in still image shooting mode, or when the imaging device is in video shooting mode.

6. The driving means further drives the lens in a rotational direction around an axis perpendicular to the optical axis, 6. The imaging device according to claim 1, wherein, when the multiple images are combined and the multiple images are captured by the imaging means, the generation means generates a rotation amount for performing the rotational drive and the tilt drive of the imaging means, generates the translational amount within a range in which the imaging means can be driven in the translational direction after the rotational drive and the tilt drive, generates a rotation amount for the tilt drive when a focal length of the photographing optical system is shorter than a predetermined focal length, generates the rotation amount for the rotational drive within a range in which the imaging means can be driven in the rotational direction after the tilt drive, generates the rotation amount for the rotational drive when the focal length is equal to or greater than the predetermined focal length, and generates the rotation amount for the tilt drive within a range in which the imaging means can be driven in the tilt direction after the rotational drive.

7. a position alignment unit that aligns the plurality of images output from the imaging unit when the plurality of images are to be synthesized; a synthesis means for synthesizing the plurality of images aligned by the alignment means; 7. The imaging device according to claim 1, further comprising:

8. the imaging means is capable of outputting a first image and a second image having different viewpoints based on light incident via an imaging optical system, a calculation means for calculating a movement vector indicating an amount of deviation between a plurality of first images obtained at a first timing and a second timing prior to the first timing, and a plurality of second images obtained at the first timing and the second timing, which have been aligned by the alignment means; a reliability detection unit that detects the reliability of the alignment by the alignment unit by integrating the movement vector obtained by the calculation unit, 8. The imaging device according to claim 7, characterized in that, when the reliability detected by the reliability detection means is lower than a predetermined threshold, the first image and the second image obtained at the first timing are not used for synthesis by the synthesis means.

9. 9. The imaging apparatus according to claim 8, wherein the calculation means forms a circulating path of the movement vector among the first images and the second images aligned by the alignment means.

10. the calculation means calculates a first motion vector from the first image obtained at the first timing to the second image obtained at the second timing, a second motion vector from the second image obtained at the first timing to the second image obtained at the second timing, a third motion vector from the second image obtained at the second timing to the first image, and a fourth motion vector from the first image obtained at the second timing to the first image obtained at the first timing, after the alignment by the alignment means; 10. The imaging apparatus according to claim 9, wherein the reliability detection means detects the reliability by integrating the first to fourth motion vectors.

11. the calculation means calculates a first motion vector from the first image obtained at the first timing to the second image after alignment by the alignment means, a second motion vector from the second image obtained at the first timing to the first image obtained at the second timing, a third motion vector from the first image obtained at the second timing to the second image, and a fourth motion vector from the second image obtained at the second timing to the first image obtained at the first timing; 9. The imaging apparatus according to claim 8, wherein the reliability detection means detects the reliability by integrating the first to fourth movement vectors.

12. the imaging means includes an imaging element having a plurality of pixels, each including a first photoelectric conversion unit and a second photoelectric conversion unit that photoelectrically convert light that has passed through different pupil regions of the photographing optical system; 12. The imaging device according to claim 8, wherein the first image corresponds to an image signal photoelectrically converted by the first photoelectric conversion unit, and the second image corresponds to an image signal photoelectrically converted by the second photoelectric conversion unit.

13. the imaging means includes a first imaging element and a second imaging element that photoelectrically convert light incident thereon via a plurality of photographing optical systems, 12. The imaging device according to claim 8, wherein the first image corresponds to an image signal photoelectrically converted by the first imaging element, and the second image corresponds to an image signal photoelectrically converted by the second imaging element.

14. an acquisition means for acquiring a first image and a second image having different viewpoints obtained based on light incident via an imaging optical system from an imaging means; a positioning unit that performs positioning between the first images obtained by the acquisition unit at a first timing and a second timing prior to the first timing, and performs positioning between the second images obtained at the first timing and the second timing; a synthesis means for synthesizing the first images and the second images aligned by the alignment means; a calculation means for calculating a movement vector indicating an amount of deviation between the plurality of first images and the plurality of second images aligned by the alignment means; a reliability detection means for detecting the reliability of the alignment by the alignment means by integrating the movement vector obtained by the calculation means, An image processing device characterized in that, when the reliability detected by the reliability detection means is lower than a predetermined threshold, the first image and the second image obtained at the first timing are not used for synthesis by the synthesis means.

15. an imaging step in which the imaging means photoelectrically converts light incident via the imaging optical system and outputs an image; a blur detection step of detecting blur of an imaging device; a driving step in which a driving unit drives the imaging unit in a translational direction on a plane perpendicular to an optical axis of the photographing optical system and in a rotational direction around the optical axis; a generating step of generating a translation amount of the translation drive and a rotation amount of the rotation drive so as to correct the shake detected in the shake detection step, A control method for an imaging device, characterized in that when multiple images output in the imaging process are synthesized and the multiple images are captured by the imaging means, the generation process generates a rotation amount for driving the imaging means in rotation, and generates the translation amount within a range in which the imaging means can be driven in translation after the rotational drive.

16. an imaging step in which the imaging means photoelectrically converts light incident via the imaging optical system and outputs an image; a blur detection step of detecting blur of an imaging device; a driving step of driving the imaging means in a translational direction on a plane perpendicular to an optical axis of the photographing optical system and in a rotational direction around the optical axis; a generating step of generating a translation amount of the translation drive and a rotation amount of the rotation drive so as to correct the shake detected in the shake detection step, A control method for an imaging device, characterized in that, in the generation process, when a plurality of images output from the imaging means are combined, when the plurality of images are photographed by the imaging means, an upper limit value of the rotation amount of the rotational drive is set higher and an upper limit value of the translation amount of the translation drive is set lower than when the combination is not performed.

17. an acquisition step in which an acquisition means acquires, from an imaging means, a first image and a second image having different viewpoints obtained based on light incident via an imaging optical system; a positioning step in which, when the first images and the second images obtained from the imaging means are combined, a positioning means performs positioning between the first images obtained at a first timing and a second timing that is earlier than the first timing in the acquisition step, and performs positioning between the second images obtained at the first timing and the second timing; a combining step of combining the first images and the second images, the first images and the second images being aligned in the alignment step, when the combining means combines the first images and the second images; a calculation step in which a calculation means calculates a movement vector indicating an amount of deviation between the plurality of first images and the plurality of second images aligned in the alignment step; a reliability detection step in which a reliability detection means detects the reliability of the alignment by the alignment means by integrating the movement vector obtained in the calculation step, An image processing method characterized in that, when the reliability detected in the reliability detection process is lower than a predetermined threshold, the first image and the second image obtained at the first timing are not used for synthesis in the synthesis process.

18. A program for causing a computer to function as each of the means of the image processing device according to claim 14.

19. A program for causing a computer to execute each step of the method for controlling an imaging apparatus according to claim 15 or 16.

Images