IMAGE PROCESSING DEVICE, IMAGE PROCESSING METHOD AND IMAGE ACCEPTATION DEVICE

Abstract

Image processing device (100) with: a processor (101) configured to generate a second image by applying one or more processes to a first image captured via an optical system (102), where, if the first image has a distortion caused by aberration of the optical system (102), and the image center coordinates of the second image and the coordinates corresponding to a center of the aberration do not match, The processor (101) generates the second image such that it has a point-symmetric distortion centered on the image center coordinates by applying a transformation process to the first image.

Description

BACKGROUND OF THE INVENTIONField of the invention

[0001] The present invention relates to an image processing apparatus, an image processing method and an image pickup apparatus. Description of related technology

[0002] It is known to perform accurate image shake correction on an image whose distortion is caused by aberration of an optical imaging system (Japanese Patent Laid-Open No. 2006-33759). Furthermore, it is known that image distortion caused by aberration can be classified into barrel distortion and pincushion distortion, and that barrel distortion appears more natural. Therefore, in Japanese Patent Laid-Open No. 2006-33759, barrel distortion is intentionally added to an image to which image shake correction has been applied.

[0003] The degree of image distortion caused by aberration in an optical imaging system depends on the distance of the object from the optical axis (image height). Therefore, when an effect simulating image distortion caused by aberration is applied, the degree of distortion is determined by assuming that the optical axis is orthogonal to the center of the image. However, when performing image shake correction on a moving image, the positional relationship between the optical axis of the optical imaging system and the center of each moving image frame changes over time due to the image shake correction.Therefore, when a barrel distortion effect is applied to motion picture frames that have undergone image shake correction by assuming that the optical axis is always orthogonal to the center of the image, the relationship between the image height and the degree of distortion changes over time. In particular, such temporal variation in distortion appears unnatural in areas with large image heights. Japanese Patent Laid-Open No. 2006-33759 makes no mention of such a problem at all.

[0004] US 2012 / 0 113 278 A1 discloses an imaging device comprising a plurality of imaging units; a correction unit that performs a correction process for images captured by a plurality of the imaging units; and a control unit that calculates a correction parameter applied to a correction process in the correction unit, wherein the correction unit performs distortion aberration correction and hand vibration correction for each of the captured images and an image characteristic adjustment correction process for adjusting characteristics between a plurality of images captured by a plurality of the imaging units. SUMMARY OF THE INVENTION

[0005] The present invention provides an image processing apparatus and an image processing method that make it possible to eliminate the problem of such a conventional technique and to produce a moving image that appears natural after image blur correction.

[0006] According to one aspect of the present invention, there is provided an image processing apparatus comprising: a processor configured to generate a second image by applying one or more processes to a first image captured via an optical system, wherein, when the first image has distortion caused by aberration of the optical system and the image center coordinates of the second image and the coordinates corresponding to a center of the aberration do not match, the processor generates the second image to have point-symmetric distortion centered on the image center coordinates by applying a transformation process to the first image.

[0007] According to another aspect of the present invention, there is provided an image processing apparatus that generates a moving image whose distortion caused by aberration of an optical system used to capture the moving image and whose image shake caused by shaking of a device have been corrected, characterized by: processing the moving image so that a distortion effect is added to each frame of the moving image, using a center position of the frame as a reference position.

[0008] According to another aspect of the present invention, there is provided an image capturing apparatus comprising: an image processing apparatus according to the present invention; and an image sensor configured to generate the first image corresponding to an optical image formed by the optical system.

[0009] According to another aspect of the present invention, there is provided an image processing method comprising: generating a second image by applying one or more processes to a first image captured via an optical system, the first image having distortion caused by aberration of the optical system, image center coordinates of the second image and coordinates corresponding to a center of the aberration do not match, and the second image having point-symmetric distortion centered on the image center coordinates is generated by applying a transformation process to the first image.

[0010] According to another aspect of the present invention, there is provided an image processing method comprising: generating a moving image whose distortion caused by aberration of an optical system used to capture the moving image and whose image shake caused by shaking of a device have been corrected, the processing comprising processing the moving image so that a distortion effect is added to each frame of the moving image, using a center position of the frame as a reference position.

[0011] According to a further aspect of the present invention, a program is provided which, when executed by an image processing device, causes the image processing device to perform a method according to the present invention.

[0012] According to one aspect of the present invention, there is provided a computer-readable storage medium storing a program according to the present invention.

[0013] Further features of the present invention will become apparent from the following description of exemplary embodiments with reference to the accompanying drawings. Each of the embodiments of the present invention described below can be implemented individually or as a combination of a plurality of the embodiments. Features of different embodiments can also be combined if necessary or if combining elements or features of individual embodiments in a single embodiment is advantageous. BRIEF DESCRIPTION OF THE DRAWINGS Fig. 1 is a block diagram showing a configuration example of an image pickup apparatus according to an embodiment of the present invention. Fig. 2 is a flowchart showing an example of control according to an embodiment of the present invention. Fig. 3 is a graph for explaining distortion correction processing and distortion addition processing according to an embodiment of the present invention. Fig. 4 is a block diagram showing a configuration example of an image shake correction amount calculation circuit 203 according to an embodiment of the present invention. Fig. 5 is a diagram for explaining effects of distortion correction processing and distortion addition processing according to an embodiment of the present invention. Fig. 6 is a block diagram showing a configuration example of an image pickup apparatus according to a second embodiment of the present invention. Fig. 7 is a block diagram showing a configuration example of an optical correction control circuit 220 according to the second embodiment of the present invention. Fig. 8 is a flowchart showing an example of control according to the second embodiment of the present invention. Fig. 9 is a diagram for explaining effects of distortion correction processing and distortion addition processing according to the second embodiment of the present invention. Fig. 10 is a graph for explaining distortion correction processing and distortion addition processing according to a third embodiment of the present invention. DESCRIPTION OF THE EMBODIMENTS

[0014] Exemplary embodiments of the present invention will now be described in detail in accordance with the accompanying drawings. In the drawings, like reference numerals are assigned to like elements, and redundant descriptions are omitted. Note that, throughout this specification, a movement of an image pickup device is referred to as "shake," and the influence of "shake" on a captured image is referred to as "image shake." A configuration applied to a digital video camera, which is one example of an image processing apparatus according to an embodiment of the present invention, will be described below, but the present invention can be applied to any electronic device capable of performing image shake correction.Such electronic devices include, but are not limited to, image capture devices, personal computers, tablet devices, mobile phones, gaming machines, disk recorders, robots, drones, and the like. First embodiment

[0015] Fig. 1 is a block diagram showing an example of a functional configuration of a video camera 100 as an example of an image processing apparatus according to an embodiment of the present invention.

[0016] According to Fig. 1, a system control circuit 101 has a programmable processor such as a CPU, as well as a RAM and a ROM. By applying or developing a program stored in the ROM to the RAM and the CPU executing the program, the system control circuit 101 controls operations of components of the video camera 100 and realizes functions of the video camera 100. Therefore, operations of the video camera 100 to be described below are mainly carried out by the system control circuit 101, unless expressly stated otherwise. It should be noted that reference numerals 201 to 203 corresponding to the blocks of the system control circuit 101 in Fig. 1, denote schematically shown functional blocks realized by the system control circuit 101, among the functional blocks of an image blur correcting device, and a description of these will be given in detail below.

[0017] A photographic lens 102 (an imaging optical system) includes a variable magnification lens, a focus lens, and the like, which are driven under the control of the system control circuit 101, and forms an object image on the imaging plane of an image sensor 103. The photographic lens 102 may or may not be detachable.

[0018] The image sensor 103 is, for example, an XY addressing type CMOS (Complementary Metal Oxide Semiconductor) image sensor. The image sensor 103 includes a plurality of pixels arranged therein, and an optical image formed by the photo lens 102 is photoelectrically converted in the pixels, thereby obtaining analog image signals.

[0019] A signal processing circuit 104 applies signal processing, such as noise reduction, A / D conversion, color interpolation, white balance adjustment, and gamma correction, to analog image signals output from the image sensor 103 to generate digital image signals (image data). The signal processing circuit 104 stores the generated image data in an image memory 105.

[0020] A distortion correction circuit 106 applies processing for correcting distortion caused by aberration of the photographing lens 102 (distortion correction processing) to image data stored in the image memory 105 and stores the resulting image data in the image memory 105. The method of distortion correction processing is not particularly limited, and a known method can be used. For example, a position is obtained that is spaced from the position of a pixel of interest in the image height direction (on a line connecting a correction center and the coordinates of the pixel of interest) at a distance that can be determined based on the image height of the pixel of interest (the distance between the correction center and the pixel of interest).There is a method for obtaining the signal level of the spaced position by performing interpolation using the signal levels of pixels in the vicinity of the spaced position and setting the signal level obtained from this interpolation as the signal level of the pixel of interest. The amount of movement (correction amount) at the time of correction takes a value that depends on the size or value of the image height and is calculated by a distortion correction amount calculation circuit 202.

[0021] The distortion correction amount calculation circuit 202 has correction amounts prepared in advance for a plurality of discrete image height values, and calculates a correction amount corresponding to the image height by interpolating prepared correction amounts when there is no correction amount corresponding to the image height of a pixel of interest. Note that a correction amount for a desired image height may be calculated using an n-order polynomial that approximates the relationship between the image height and the correction amount instead of interpolation. Note that not only the intersection point between the optical axis of the photographing lens 102 and an image but also any coordinates in the image can be designated as a correction center that determines the size of an image height when calculating a correction amount.A correction center is provided by the distortion correction amount calculation circuit 202 to be described below.

[0022] An image shake correction circuit 107 corrects image shake of an image in which distortion has been corrected by the distortion correction circuit 106, the image shake being caused by a shake of the video camera 100. For example, the image shake correction circuit 107 crops / uncrops an image in which distortion has been corrected by the distortion correction circuit 106 and reads it from the image memory 105, and changes the cropping / uncropping position of the image to cancel a shake of the video camera 100. Note that the correction is not limited to correction of translational components of a shake performed by changing the cropping / uncropping position, and a configuration may be adopted in which rotational components around the optical axis of a shake are corrected by affine transformations, or tilt / angle corrections are performed.Skew components of camera shake (keystone distortion of an image) are corrected by projection conversion. The image shake correction circuit 107 stores an image in which image shake has been corrected in the image memory 105.

[0023] A distortion addition circuit 108 reads out from the image memory 105 an image in which distortion and image blur have been corrected by the distortion correction circuit 106 and the image blur correction circuit 107, respectively, adds the effect of distortion caused by the photographing lens 102 to this image, and stores the image in the image memory 105. The distortion addition circuit 108 can add a distortion effect to an image by obtaining, as the signal level of a pixel of interest, a signal level at a position spaced from the position of the pixel of interest in the direction opposite to the distortion correction processing. In addition, the degree of distortion depends on the degree of distortion.The degree of distortion added by the distortion addition circuit 108 depends on the distance between a pixel of interest and a correction center, but a configuration is adopted in which any coordinate position in an image output from the image blur correction circuit 107 can be determined as a correction center. A correction center is obtained by the distortion correction amount calculation circuit 202, which will be described below.

[0024] A recording control circuit 111 writes moving image data, still image data, metadata, and the like stored in the image memory 105 to or reads them from a recording medium 112. The recording medium 112 may be a non-volatile memory, a magnetic recording medium, an optical recording medium, or the like, and the recording medium 112 may be removable or replaceable.

[0025] A display control circuit 109 causes a display device 110, such as a liquid crystal display (LCD), to display an image to be displayed based on moving image data or still image data stored in the image memory 105, a GUI screen such as a menu screen, and the like. In this embodiment, the recording control circuit 111 writes an image to which a distortion effect has been added by the distortion addition circuit 108 onto the recording medium 112, and the display control circuit 109 causes the display device 110 to display the image to which the distortion effect has been added by the distortion addition circuit 108.

[0026] An angular velocity sensor 114 generates shake signals representing shake of the video camera 100. The angular velocity sensor 114 is located on a plane orthogonal to the optical axis, and two straight lines orthogonal to each other serve as detection axes. An A / D converter 115 samples the shake signals generated by the angular velocity sensor 114, converts the sampled shake signals into digital data, and supplies the digital data as shake data to the image shake correction amount calculation circuit 203.

[0027] An optical parameter calculation circuit 201, the distortion correction amount calculation circuit 202, and the image blur correction amount calculation circuit 203 may be constituted by hardware equipped with dedicated arithmetic circuits to achieve their functions, or may be configured to be realized by the programmable processor of the system control circuit 101 executing programs. In this embodiment, a description will be given assuming that the optical parameter calculation circuit 201, the distortion correction amount calculation circuit 202, and the image blur correction amount calculation circuit 203 are constituted by hardware equipped with dedicated arithmetic circuits to achieve their functions.

[0028] The optical parameter calculation circuit 201 calculates optical parameters such as a focal length or an object distance of the photographing lens 102, and supplies the optical parameters to the distortion correction amount calculation circuit 202 and the image blur correction amount calculation circuit 203.

[0029] The distortion correction amount calculation circuit 202 supplies correction data and a correction means position for distortion correction and distortion addition to the distortion correction circuit 106 and the distortion addition circuit 108 based on optical parameters.

[0030] The image shake correction amount calculation circuit 203 supplies a correction amount for correcting image shake caused by shake of the video camera 100 to the image shake correction circuit 107 based on shake data supplied from the A / D converter 115.

[0031] In the configuration according to Fig. 1, the distortion correction circuit 106, the image shake correction circuit 107, the distortion addition circuit 108, the optical parameter calculation circuit 201, the distortion correction amount calculation circuit 202, the image shake correction amount calculation circuit 203, and the system control circuit 101 constitute the image shake correction device.

[0032] A conversion lens 150 is an auxiliary lens that can be detached from the photo lens 102. The system control unit 101 can detect whether a conversion lens 150 is attached or not, for example, according to the state of a switch 151 provided on the photo lens 102.

[0033] Next, an image blur correction process according to this embodiment will be described with reference to the flowchart of Fig. 2. Here, image shake correction processing for an image of a screen is described. Therefore, for example, as in the case where the display device 110 operates as an EVF (in a shooting standby state or during shooting of a moving image to be recorded), when image shake correction is applied to a moving image, the processing shown in the flowchart of Fig. The process shown in Figure 2 is performed repeatedly. Note that image shake correction does not necessarily need to be applied to all moving image frames or frames.

[0034] In step S101, the optical parameter calculation circuit 201 calculates optical parameters used to calculate distortion characteristics of the photographic lens 102. For example, the optical parameter calculation circuit 201 calculates the focal length of the photographic lens 102 from position information of the variable magnification lens acquired by the photographic lens 102. The optical parameter calculation circuit 201 also calculates an object distance (focus distance) from position information of the focus lens of the photographic lens 102. Note that these optical parameters can be acquired, for example, by referring to a table without performing any calculation.

[0035] In step S102, the distortion correction amount calculation circuit 202 calculates correction data used for the distortion correction performed by the distortion correction circuit 106. In Fig. 3, 31 and 32 show examples of distortion characteristics of the photographing lens 102, where the horizontal axis denotes the image height and the vertical axis denotes the distortion rate. A distortion rate represents a rate of change from an image height position at which an image is originally to be formed when there is no distortion. The sign of the distortion rate is positive when the direction in which the image formation position changes is a direction in which the image height increases, and is negative when the direction in which the image formation position changes is a direction in which the image height decreases. Distortion is observed as "pincushion distortion" when the distortion rate is positive, and distortion is observed as "barrel distortion" when the distortion rate is negative.

[0036] The distortion correction amount calculation circuit 202 calculates correction data according to which distortion of an image caused by aberration of the photographing lens 102 is corrected as much as possible to increase the accuracy of image shake correction performed by the image shake correction circuit 107. Therefore, correction data is ideally generated so that the characteristics of the correction data are inverse characteristics of the distortion characteristics of the photographing lens 102.

[0037] The distortion characteristics of the photo lens 102 change according to the optical parameters (here, focal length and object distance) of the photo lens 102. Therefore, in this embodiment, the distortion characteristics of the photo lens 102 are stored for each combination of discrete optical parameters, for example, in the ROM of the system control circuit 101. The distortion correction amount calculation circuit 202 then reads from the ROM the distortion characteristics corresponding to combinations that are close to a combination of actual optical parameters, performs interpolation, and thereby calculates distortion characteristics corresponding to the combination of actual optical parameters. Subsequently, the distortion correction amount calculation circuit 202 calculates inverse characteristics of the calculated distortion characteristics as a set of correction data corresponding to different image heights.Note that a configuration may be adopted in which the distortion correction amount calculation circuit 202 calculates correction data only for discrete image heights, as shown in Fig. 32 of Fig. 1. Fig. 3, and correction data for a different image height is acquired by interpolation performed by the distortion correction circuit 106. The distortion correction amount calculation circuit 202 supplies the correction data to the distortion correction circuit 106.

[0038] In step S103, the distortion correction amount calculation circuit 202 calculates a correction center used for distortion correction in the distortion correction circuit 106. The distortion correction circuit 106 corrects distortion of an image caused by aberration of the photographing lens 102. Therefore, the distortion correction amount calculation circuit 202 calculates, as a correction center, image coordinates corresponding to the intersection point between the optical axis of the photographing lens 102 and the image sensor 103 and supplies the calculated image coordinates to the distortion correction circuit 106. The image coordinates of the intersection point between the optical axis of the photographing lens 102 and the image sensor 103 may be stored in advance in the ROM of the system control circuit 101 based on a design value.Additionally, a value corrected using the amount of deviation from a design value that occurred during product manufacturing can be stored. Note that a design value that places the image coordinates of the intersection point between the optical axis and the image sensor at the center of the image is a general design value.

[0039] In step S104, the image shake correction amount calculation circuit 203 calculates an image shake correction amount based on shake data supplied from the angular velocity sensor 114 via an A / D conversion circuit, and supplies the image shake correction amount to the image shake correction circuit 107. The angular velocity sensor 114 sets at least two axes as detection axes, namely, a horizontal rotation axis (yaw) and a vertical rotation axis (pitch), and an image shake correction amount is calculated for each of the detection axes. However, processing for calculating an image shake correction amount does not depend on a detection axis, and therefore, for ease of description and understanding, processing for calculating an image shake correction amount using a detection axis will be described below.

[0040] Fig. 4 is a block diagram showing the internal configuration of the image shake correction amount calculation circuit 203. A high-pass filter (HPF) 204 is applied to shake data (angular velocity data) supplied from the A / D converter 115 to remove low-frequency components such as DC components.

[0041] A framing control circuit 208 determines, from an output of an integrator 206, whether a framing or image adjustment operation (panning or tilting) is being intentionally performed by the user. If it is determined that a framing operation is being performed, the framing control circuit 208 reduces the influence of angular velocity components related to the framing operation on angular displacement data calculated by the integrator 206. A detailed description will be given below.

[0042] The integrator 206 integrates the supplied angular velocity data and converts the integrated angular velocity data into angle data. The integrator 206 also outputs angular displacement data, which is obtained by multiplying the angle data by a predetermined coefficient, as a correction amount used by the image shake correction circuit 107.

[0043] Specifically, correction amounts Tx and Ty on the image plane are calculated for a shake angle θy of yaw and a shake angle θp of pitch using the following expressions. Tx=f×tanθy / P Ty=f×tanθp / P

[0044] (Note that f is a focal length [mm] and P is a pixel size [mm]. Here, a pixel size is the size of one side of a square pixel.)

[0045] A limiter 207 limits angular displacement data to a predetermined upper limit.

[0046] Here, operations of the framing control circuit 208 will be further described. When the angular displacement data output by the integrator 206 exceeds a predetermined threshold, the framing control circuit 208 determines that a framing operation is being performed. This threshold is smaller than the upper limit set by the limiter 207 and is a value determined in advance by conducting experiments or the like.

[0047] When it is determined that a framing operation is being performed, the framing control circuit 208 may reduce the components of the framing operation included in camera shake by: Increasing the cutoff frequency of the HPF 204 or Shortening a time constant of a low-pass filter (LPF) calculation included in an integration operation performed by the integrator 206, compared to the case where it is not determined that a framing operation is carried out.

[0048] Alternatively, a configuration may be adopted in which the framing control circuit 208 causes a subtractor 205 to subtract an offset from angular velocity data output by the HPF 204, thereby reducing the components of the framing operation included in the camera shake.

[0049] Such control of the framing control circuit 208 enables excellent image shake correction even when an intentional framing operation such as panning or tilting is performed by the user.

[0050] Returning to Fig. 2, in step S105, the distortion correction amount calculation circuit 202 determines whether the focal length of the photographing lens 102 is smaller than a threshold value or not, and the flow proceeds to step S106 if it is determined that the focal length is smaller than the threshold value, and the flow proceeds to step S109 if it is not determined that the focal length is smaller than the threshold value. Note that this determination in step S105 is not necessary, and the flow may proceed from step S104 to step S106 regardless of the focal length of the photographing lens 102.

[0051] Typically, it is recognized or observed that an image captured using a (super) wide-angle lens such as a fisheye lens has aberration-caused distortion as a characteristic. In addition, when a large amount of aberration-caused distortion is corrected, volume distortion occurs in which the image of an object reproduced in a peripheral area of ​​an image appears enlarged, and a viewer therefore feels that something is wrong. Accordingly, it is preferable that aberration-caused image distortion be absent for image shake correction, but with respect to an image captured with a wide-angle lens, it is indeed normal for a certain degree of aberration-caused image distortion to be present.

[0052] Therefore, in this embodiment, the distortion adding circuit 108 adds a distortion effect when the focal length of the photo lens 102 is smaller than the threshold value, and does not add a distortion effect when the focal length of the photo lens 102 is greater than or equal to the threshold value. Here, the threshold value may be a visually desirable value when a distortion similar to an aberration is added to an image.

[0053] In step S106, the distortion correction amount calculation circuit 202 calculates the characteristics of a distortion effect used by the distortion addition circuit 108. In this embodiment, a distortion effect added by the distortion addition circuit 108 is calculated based on the distortion characteristics of a photographing lens (i.e., the photographing lens 102) used to capture an image subjected to image blur correction. For example, when the maximum distortion rate of the photographing lens 102 is -10%, as shown in Fig. 31 of Fig. As shown in Figure 3, distortion addition data for adding distortion (barrel distortion) whose maximum distortion rate is approximately equal to -3% is calculated. This distortion addition data can be obtained by multiplying distortion correction data shown in Figure 32 by Fig. 3 are calculated with a magnification of -3 / 10. Note that, here, it is assumed that a distortion effect of approximately 30% of the degree of distortion of the photo lens 102 is added, but the degree of distortion effect added by the distortion adding circuit 108 can be appropriately set within a value range not exceeding the intensity of distortion caused by aberration of the photo lens 102. Alternatively, to simplify processing, pre-prepared default distortion adding data may be read out regardless of the photo lens 102.

[0054] In step S107, the distortion correction amount calculation circuit 202 calculates a correction center used when the distortion addition circuit 108 adds a distortion effect, supplies the correction center to the distortion addition circuit 108, and advances the flow to step S108. The distortion correction circuit 106 uses, as a correction center, image coordinates corresponding to the intersection point between the optical axis and the image sensor to correct image distortion caused by aberration of the photographing lens 102. On the other hand, in an image subjected to image shake correction, due to a cropping / cropping position being changed by the image shake correction circuit 107, the image coordinates corresponding to the intersection point between the optical axis and the image sensor change over time.

[0055] Therefore, when the image coordinates are set according to the intersection point between the optical axis and the image sensor as a correction center when adding a distortion effect, the correction center in the image moves according to the change in the image shake correction amount. As shown in Figure 33 of Fig. As shown in Figure 3, a distortion effect has a distortion rate that depends on the image height (i.e., the distance between the correction center and a target pixel). Therefore, as the correction center moves over time, a distortion rate for the same image coordinates in moving image frames subjected to image distortion correction also changes over time, resulting in a moving image that appears unnatural. Therefore, the distortion correction amount calculation circuit 202 calculates center coordinates of an image subjected to image blur correction (cropping) and supplies the center coordinates as a correction center upon addition of a distortion effect to the distortion addition circuit 108.

[0056] On the other hand, if it is not determined in step S105 that the focal length of the photographing lens 102 is smaller than the threshold value, the distortion correction amount calculation circuit 202 sets the distortion adding circuit 108 not to add a distortion effect in step S109 and advances the process to step S108.

[0057] In step S108, the distortion correction circuit 106 performs distortion correction on image data stored in the image memory 105 based on distortion correction data and a correction center provided from the distortion correction amount calculation circuit 202. In addition, the image shake correction circuit 107 performs image shake correction on the image data output by the distortion correction circuit 106 based on an image shake correction amount provided from the image shake correction amount calculation circuit 203.When a setting for adding a distortion effect has been made, the distortion adding circuit 108 adds a distortion effect to the image data output by the image shake correcting circuit 107 based on distortion adding data and a correction center supplied from the distortion correction amount calculating circuit 202, and stores the image data in the image memory 105.

[0058] Note that here, the distortion addition circuit 108 is set not to add a distortion effect when it is determined that the focal length of the photographing lens 102 is shorter than the threshold. However, distortion addition data with which a distortion effect is not added may be supplied from the distortion correction amount calculation circuit 202 to the distortion addition circuit 108.

[0059] With reference to Fig. 5, image processing in the distortion correction circuit 106, the image blur correction circuit 107, and the distortion addition circuit 108 will be further described here.

[0060] In Fig. 5 schematically shows 51 a distortion of an image captured by the image sensor 103 and stored in the image memory 105. Here, barrel distortion caused by aberration of the photographing lens 102 has occurred, centered at image coordinates (x, y) corresponding to the intersection point between an optical axis O and the image sensor (which is assumed here to coincide with the center coordinates of the image).

[0061] At the time of distortion correction, image shake correction was not performed, and therefore the image coordinates corresponding to the intersection point between the optical axis O and the image sensor are still (x, y). Therefore, the distortion correction amount calculation circuit 202 supplies the coordinates (x, y) as a correction center to the distortion correction circuit 106. Due to distortion correction performed by the distortion correction circuit 106, a Fig. 5 image distortion corrected (eliminated) (52, Fig. 5).

[0062] The image shake correction circuit 107 performs image shake correction on the image in 52 of Fig. 5 in which distortion has been corrected. The image shake correction circuit 107 determines a cropping / cropping area to eliminate image shake resulting from shake of the video camera 100, based on an image shake correction amount calculated by the image shake correction amount calculation circuit 203. The image shake correction circuit 107 outputs the image of the determined cropping / cropping area as an image subjected to image shake correction. Unless the image shake correction amount is equal to 0, the image coordinates (x, y) and the center coordinates of the image subjected to image shake correction (cropping area) do not match (53, Fig. 5).

[0063] The distortion adding circuit 108 adds a distortion effect to the image subjected to image blur correction (54, Fig. 5). The image coordinates (x, y) corresponding to the intersection point between the optical axis O and the image sensor do not coincide with coordinates (x', y') of a center O' of the image subjected to image shake correction. Therefore, the distortion correction amount calculation circuit 202 supplies the center coordinates (x', y') of the image subjected to image shake correction, as a correction center upon addition of a distortion effect, to the distortion addition circuit 108. As a result, it is possible to add a point-symmetric distortion effect in which the degree of distortion added to pixels equally spaced from the center of the image subjected to image shake correction is the same, as shown in FIG. 54 of Fig. 5. Therefore, a distortion rate in a peripheral area of ​​a moving image no longer changes with time, and it is possible to produce a moving image that does not appear unnatural when viewed.

[0064] Note that, in this embodiment, in order to clearly describe the gist of the invention, a configuration was described in which the distortion correction circuit 106, the image shake correction circuit 107, and the distortion addition circuit 108 independently apply distortion correction processing, image shake correction processing, and distortion effect addition processing. However, a configuration may be adopted in which the distortion correction processing, the image shake correction processing, and the distortion effect addition processing are combined into coordinate conversion based on a correction amount and a correction center in each processing, and applied to image data stored in the image memory 105.With such a configuration, it is possible to perform distortion correction processing, image shake correction processing, and distortion effect addition processing at the same time, and correction processing of image data is performed at one time. Alternatively, a configuration may be adopted in which coordinate conversion for correcting the distortion characteristics of a photographic lens and coordinate conversion for adding a distortion effect are combined, and, according to the result of this combination, distortion correction processing and distortion effect addition processing are applied to image data, and then image cropping / cropping for image shake correction is performed.

[0065] As described above, in this embodiment, transformation processing is performed on an image including distortion caused by aberration of a photographing lens to correct the distortion caused by aberration and image blur, and an image to which a distortion effect has been added is generated using the center coordinates of the image as a correction center. Therefore, even in the case of adding a distortion effect to moving image frames, it is possible to suppress the distortion rate in a peripheral area from changing with time and to generate a moving image that does not appear unnatural. In addition, accurate image blur correction can be achieved by performing image blur correction on an image in which a distortion caused by aberration has been added.The distortion caused by aberrations in a photographic lens has been corrected. Additionally, it is possible to create a moving image that exhibits the effect of a wide-angle shot by adding a distortion effect when the focal length of a photographic lens is shorter than a threshold. Second embodiment

[0066] Next, a second embodiment of the present invention will be described. The first embodiment is directed to an image pickup device that changes a cropping / cropping area of ​​a captured image and performs image shake correction, but this embodiment differs from the first embodiment in that this embodiment is directed to an image pickup device that performs image shake correction by driving an optical element.

[0067] Fig. 6 is a block diagram showing an example of a functional configuration of a video camera 100' according to this embodiment. In Fig. 6 are the same reference numerals as those in Fig. 1 is assigned to the same components as in the first embodiment, and a description regarding components described in the first embodiment will be omitted.

[0068] The video camera 100' includes components for optically correcting image shake instead of the components (the image shake correction circuit 107 and the image shake correction amount calculation circuit 203) for electronically correcting image shake among the components of the first embodiment. Specifically, the photo lens 102 is provided with an optical correction system 116 and components (118 to 121) for driving the optical correction system 116, and an optical correction control circuit 220 that controls the components. Here, the optical correction control circuit 220 may be formed by hardware equipped with dedicated arithmetic circuits for achieving respective functions, or configured to be implemented by a CPU of a system control circuit 101 executing programs. It should be noted that according to Fig. 6 the optical correction system 116 is controlled, but alternatively or additionally an image sensor 103 and the entirety of the photo lens or the photo objective 102 can be controlled.

[0069] The optical correction system 116 is, for example, a shift lens and can move to change the direction of the optical axis of the photo lens 102 (the intersection point between the optical axis and the image sensor). The optical correction control circuit 220 moves the optical correction system 116 to eliminate shake of the video camera 100', thereby forming a subject image in which image shake has been corrected on the imaging plane of the image sensor 103.

[0070] A position detector 117 outputs a voltage corresponding to the position of the optical correction system 116. This voltage is amplified by the amplifier 118, then digitized by the A / D converter 119, and supplied as position data to the optical correction control circuit 220.

[0071] The optical correction control circuit 220 calculates camera shake correction data based on shake data supplied from an A / D converter 115. The camera shake correction data represents a target position to which the optical correction system 116 is to be controlled. The optical correction control circuit 220 drives the optical correction system 116 based on the difference between the position of the optical correction system 116 supplied by the A / D converter 119 and the target position. The configuration and operations of the optical correction control circuit 220 are described below.

[0072] A PWM circuit 120 modulates data (deviation data) output by the optical correction control circuit 220, representing the difference between the target position and the current position, into a waveform (PWM waveform) for changing the duty ratio of a pulse wave, and supplies the modulated data to a motor drive circuit 121. The motor drive circuit 121 drives a motor 122, for example, a voice coil type motor, based on the PWM waveform supplied by the PWM circuit 120. The optical correction system 116 moves according to a drive direction and a drive amount of the motor 122.

[0073] Subsequently, deviation data is calculated, and the optical correction system 116 is driven based on the position and wobble data of the optical correction system 116 that has moved. In this way, the optical correction control circuit 220 performs feedback control of the position of the optical correction system 116 so that the difference between the drive target position and the position data is small. Accordingly, the optical correction system 116 is driven to follow the drive target position.

[0074] Fig. Fig. 7 is a block diagram schematically showing a processing of the optical correction control circuit 220, wherein the same reference numerals as those in Fig. 4 blocks are assigned, which perform processing similar to that of the image shake correction amount calculation circuit 203 of the first embodiment. A sensitivity calculation circuit 221 converts angular velocity data into a movement amount of the optical correction system 116 using focal length information acquired by an optical parameter calculation circuit 201. An output of an integrator 206, which integrates a movement amount subjected to framing control similar to the first embodiment, represents a movement amount (a target position) of the optical correction system 116 depending on the magnitude and direction of the shake. A limiter 207 limits a movement amount to be within a movement range of the optical correction system.

[0075] A subtractor 222 detects the difference between the target position of the correction optical system 116 output by the limiter 207 and the current position of the correction optical system 116 output by the A / D converter 119, and supplies the difference as deviation data to a control filter 223. The control filter 223 applies signal processing, such as amplification and phase compensation, to the deviation data and then supplies the processed deviation data to the PWM circuit 120.

[0076] An image plane conversion circuit 224 converts position information of the correction optical system 116 output by the A / D converter 119 into a movement amount and a movement direction, assuming that the position of the optical axis position on the image sensor when the correction optical system 116 is not moving is a reference position, and supplies the movement amount and the movement direction to a distortion correction amount calculation circuit 202. The position of the optical axis position on the image sensor when the correction optical system 116 is not moving generally corresponds to the center position of the image sensor 103 or the center position of an image captured by the image sensor, as described below.

[0077] Note that processing for one detection axis of an angular velocity sensor 114 has also been described here, but similar processing is performed for another / different detection axis.

[0078] Such processing makes it possible to optically correct image blur caused by shaking of the video camera 100'.

[0079] Next, referring to the flowchart of Fig. 8, an image blur correction process according to this embodiment is described. In Fig. 8 are the same reference numerals as those in Fig. 2 is assigned to the same processing as in the first embodiment. This embodiment is similar to the first embodiment, except that there is no processing for calculating an image shake correction amount (step S104) because image shake correction is performed optically in this embodiment, and contents of processing for calculating a correction center (steps S201 and S202) are different.

[0080] A description will be given of processing for calculating a correction center performed by a distortion correction circuit 106 in step S201. Also in this embodiment, distortion correction performed by the distortion correction circuit 106 is processing for correcting distortion in a captured image caused by aberration of the photographing lens 102. Therefore, it is necessary to perform distortion correction using image coordinates corresponding to the intersection point between the optical axis of the photographing lens 102 and the image sensor 103 as a correction center.

[0081] In the first embodiment, in a captured image (a moving image frame) to be subjected to distortion correction, the image coordinates corresponding to the intersection point between the optical axis of the photographing lens 102 and the image sensor 103 are fixed. Therefore, even if a deviation from a design value due to an assembly error is taken into account, a correction center does not change during distortion correction.

[0082] However, in this embodiment, an image subjected to optical image blur correction is captured, and therefore the intersection point between the optical axis of the photographing lens 102 and the image sensor 103 changes according to the position of the optical correction system 116. Therefore, it is necessary to obtain a correction center of the distortion correction performed by the distortion correction circuit 106 based on the position of the optical correction system 116 and supply the correction center to the distortion correction circuit 106.

[0083] As stated with reference to Fig. As described in Figure 7, the optical correction control circuit 220 supplies to the distortion correction amount calculation circuit 202 a value obtained by converting the current position of the correction optical system 116 into a movement amount using the position of the optical axis on the image sensor 103 when the correction optical system 116 is not moving as a reference position. The distortion correction amount calculation circuit 202 can calculate image coordinates corresponding to the intersection point between the current optical axis of the photographing lens 102 and the image sensor 103 by applying the optical axis movement amount obtained from the current position of the correction optical system 116 to the initial optical axis position.Here, the initial position of the optical axis represents image coordinates corresponding to the intersection point between the optical axis of the photographing lens 102 and the image sensor 103 when the optical correction system 116 is stationary. The distortion correction amount calculation circuit 202 supplies the image coordinates of the correction center calculated in this way to the distortion correction circuit 106.

[0084] In this embodiment, similar to the first embodiment, the center coordinates of an image last output when a distortion addition circuit 108 adds a distortion effect are used as a correction center. Note that in the case of adding a distortion effect to the entirety of a distortion-corrected image, a correction center is located at fixed coordinates. Also, in the case of performing cropping / cropping, the correction center is located at fixed coordinates when the center of a cropping area is the same as the center of the distortion-corrected image.

[0085] Other processing may be the same as in the first embodiment except that image shake correction is not performed in step S108, and therefore a description thereof is omitted.

[0086] Fig. Fig. 9 schematically shows image processing in the distortion correction circuit 106 and the distortion addition circuit 108, similar to Fig. 5.

[0087] In Fig. 9 schematically shows distortion of an image captured by the image sensor 103 and stored in an image memory 105. Here, barrel distortion has occurred in the image caused by aberration of the photo lens 102, centered at image coordinates (x, y) corresponding to the intersection point between an optical axis O and the image sensor. Because the optical correction system 116 has moved, the image coordinates (x, y) and the center coordinates of the image do not match.

[0088] The distortion correction amount calculation circuit 202 supplies the coordinates (x, y) as a correction center to the distortion correction circuit 106. Due to a distortion correction performed by the distortion correction circuit 106, a distortion correction value shown in 91 of Fig. 9 corrected (eliminated) image distortion (92, Fig. 9). At this time, an image is captured in which image blur and distortion caused by aberration have been corrected.

[0089] Subsequently, the distortion correction amount calculation circuit 202 supplies center coordinates (x', y') of the image as a correction center when adding a distortion effect to the distortion adding circuit 108. Accordingly, the distortion adding circuit 108 can add a point-symmetric distortion effect in which the size of a distortion added to pixels equally spaced from the center of the image is the same, as shown in Fig. 93 of Fig. 9. Therefore, a distortion rate in a peripheral area of ​​a moving image no longer changes with time, and it is possible to produce a moving image that does not appear unnatural when viewed.

[0090] As described above, according to this embodiment, even in the case of optically correcting image blur, distortion caused by aberration of an optical system is corrected by performing transformation processing on an image including the distortion caused by aberration, and an image is generated to which a distortion effect has been added using the center coordinates of the image as a correction center. Accordingly, a similar effect to that of the first embodiment is achieved. Third embodiment

[0091] Next, a third embodiment of the present invention will be described. This embodiment can be achieved in combination with the first embodiment and the second embodiment.

[0092] In the first and second embodiments, a configuration is adopted in which, when the focal length of the photographing lens 102 is less than a threshold value, a distortion effect is added, and, when the focal length of the photographing lens 102 is greater than or equal to the threshold value, a distortion effect is not added.

[0093] On the other hand, there are cases where, in a video camera whose lens is not removable or replaceable, and the like, a conversion lens 150 ( Fig. 1) attached to a photographic lens 102 to change the focal length of the photographic lens 102. In this embodiment, control will be described that is performed when a detachable conversion lens (or an accessory lens) is attached to the photographic lens 102 of a video camera 100 or 100'.

[0094] In Fig. Fig. 10 shows 1001 a distortion characteristic 301 of the photo lens 102 alone and a distortion characteristic 302 of the entire optical system when a wide-angle conversion lens is attached to the photo lens 102 (combination of photo lens and wide-angle conversion lens). A wide-angle conversion lens enables a wider angle shot than with the photo lens 102, but causes greater barrel distortion. In the example of 1001 of Fig. 10, the maximum distortion rate of the photographic lens 102 alone is -10%, while the maximum distortion rate in the case where a wide-angle conversion lens is attached is -15%, and barrel distortion is stronger.

[0095] A system control circuit 101 can electrically or mechanically detect whether a wide-angle conversion lens is attached, for example, by using a switch 151 ( Fig. 1). Alternatively, a configuration may be adopted in which a setting as to whether or not a wide-angle conversion lens is attached can be changed by the user via a menu screen, and the system control circuit 101 determines whether or not a wide-angle conversion lens is attached according to the setting content.

[0096] When the system control circuit 101 determines that a conversion lens is attached, a distortion correction amount calculation circuit 202 calculates correction data for correcting the distortion characteristic 302 of the entire optical system including the conversion lens. For example, the distortion correction amount calculation circuit 202 calculates correction amounts corresponding to a plurality of discrete image heights according to an inverse characteristic of the distortion characteristic 302 in 1001 of Fig. 10, which is indicated by reference numeral 304 in 1002 of Fig. 10 is designated.

[0097] Next, calculation of distortion addition data when a wide-angle conversion lens is attached will be described. Attaching a wide-angle conversion lens increases the maximum distortion rate of the entire optical system from 10% to 15%. Therefore, similarly, when it is determined that a wide-angle conversion lens is attached, the distortion effect added by a distortion addition circuit 108 is greater than the distortion effect added when it is not determined that a wide-angle conversion lens is attached.

[0098] For example, the characteristic of distortion addition data when it is determined that a wide-angle conversion lens is not attached is indicated by reference numeral 305 in 1003 of Fig. 10. In this case, when it is determined that a wide-angle conversion lens is attached, the distortion correction amount calculation circuit 202 generates distortion addition data with an increased distortion maximum distortion rate. For example, the distortion correction amount calculation circuit 202 may generate distortion addition data with a characteristic indicated by reference numeral 306 in the diagram shown in FIG. 1003 of Fig.10. Here, it is assumed that the distortion correction amount calculation circuit 202 generates distortion addition data for adding the distortion effect 306 having a characteristic for canceling the difference between the correction amount characteristics 303 and 304. In other words, the distortion correction amount calculation circuit 202 generates distortion addition data for adding a distortion effect equivalent to distortion caused by a conversion lens or an accessory lens. This makes it possible to prevent the angle of view widened by a wide-angle conversion lens from being narrowed as a result of performing distortion correction.It is also possible to reduce the influence of volume distortion that increases as a result of attaching a wide-angle conversion lens and to produce a natural moving image.

[0099] As described above, according to this embodiment, even in the case where a conversion lens or an accessory lens is attached and the characteristics of distortion of the entire optical system change, it is possible to obtain an effect similar to those of the first and second embodiments. Further examples

[0100] The present invention has been described above based on exemplary embodiments, but the present invention is not limited to these specific embodiments, and various modifications can be made within the scope of the invention. Furthermore, in the aforementioned embodiments, a configuration in which device shake is detected by an angular velocity sensor was described, but a different configuration may be used for detection. For example, a shake amount may be calculated from acceleration detected by an acceleration sensor, or a shake amount of the device may be calculated by detecting motion information from a captured image.

[0101] One or more embodiments of the present invention may also be implemented by a computer of a system or apparatus that retrieves and executes computer-executable instructions (e.g., one or more programs) recorded on a storage medium (which may also be more fully referred to as a "non-transitory computer-readable storage medium") to perform the functions of one or more of the above-described embodiments, and / or one or more circuits (e.g.,an application-specific integrated circuit (ASIC) for performing the functions of one or more of the above-described embodiments, and by a method performed by the computer of the system or device, for example, by retrieving and executing the computer-executable instructions from the storage medium to perform the functions of one or more of the above-described embodiments and / or controlling the one or more circuits to perform the functions of one or more of the above-described embodiments. The computer may have one or more processors (e.g., central processing unit (CPU), microprocessing unit (MPU)) and may include a network of separate computers or separate processors to retrieve and execute the computer-executable instructions.The computer-executable instructions may be provided to the computer, for example, from a network or the storage medium. The storage medium may include, for example, one or more of a hard disk, random access memory (RAM), read-only memory (ROM), distributed computing system memory, an optical disk (such as a compact disc (CD), a digital versatile disc (DVD), or a Blu-ray Disc (BD)™), a flash memory device, a memory card, and the like.

[0102] While the present invention has been described with reference to exemplary embodiments, it is to be understood that the invention is not limited to the disclosed exemplary embodiments.

[0103] An image processing device is provided that generates a natural-looking image even from an image in which image blur has been corrected and distortion has been intentionally added. A processor of this image processing device performs transformation processing on an image to generate an image with point-symmetric distortion centered on the center coordinates of the image, even for an image in which the center position of an aberration of an optical system and the center position of the image do not match.

Claims

[1] Image processing device (100) comprising: a processor (101) configured to generate a second image by applying one or more processes to a first image captured via an optical system (102), wherein, if the first image has a distortion caused by aberration of the optical system (102), and the image center coordinates of the second image and the coordinates corresponding to a center of aberration do not match, the processor (101) generates the second image to have a point-symmetric distortion centered at the image center coordinates by applying a transformation process to the first image. [2] The image processing apparatus (100) according to claim 1, wherein the processor (101) is configured to apply, as the transformation process, a process for correcting image distortion caused by aberration of the optical system (102) and a process for adding a distortion effect with a distortion rate depending on a distance from the center coordinates of the second image. [3] The image processing apparatus (100) according to claim 2, wherein the processor (101) is configured to apply the process for correcting image distortion caused by aberration of the optical system (102) and subsequently apply the process for adding a distortion effect. [4] The image processing apparatus (100) according to claim 2, wherein the processor (101) is configured to apply a combination of the process for correcting image distortion caused by aberration of the optical system (102) and the process for adding a distortion effect. [5] The image processing apparatus (100) according to any one of claims 2 to 4, wherein the first image is an image in which image blur has been corrected by an image blur correcting apparatus (106, 107, 108, 201, 202, 203, 101) configured to optically correct image blur. [6] The image processing apparatus (100) according to claim 5, wherein the image shake correcting apparatus (106, 107, 108, 201, 202, 203, 101) is configured to optically correct an image shake by driving the entire optical system (102) or an optical correction system and / or an image sensor (103) according to a shake of a device. [7] The image processing apparatus (100) according to claim 2, wherein the one or more processes include a process of cutting out a region according to a shaking of the apparatus. [8] The image processing apparatus (100) according to claim 7, wherein the processor (101) is configured to apply, to the first image, the process for correcting image distortion caused by aberration of the optical system (102), the process for cutting out a region according to a shake of the apparatus, and the process for adding a distortion effect in this order. [9] The image processing apparatus (100) according to claim 7, wherein the processor (101) is configured to apply, to the first image, the process for correcting image distortion caused by aberration of the optical system (102) and the process for adding a distortion effect, and then apply the process for cutting out a region according to a shake of the apparatus. [10] The image processing apparatus (100) according to claim 7, wherein the processor (101) is configured to apply, to the first image, the process for correcting image distortion caused by aberration of the optical system (102), the process for cutting out a region according to a shake of the apparatus, and the process for adding a distortion effect at the same time. [11] The image processing apparatus (100) according to any one of claims 2 to 10, wherein the processor (101) is configured to set image coordinates corresponding to a crossing point between an optical axis of the optical system (102) and an image sensor (103) as coordinates corresponding to a center of aberration. [12] Image processing device (100) according to one of claims 2 to 11, additionally comprising: a detector configured to detect that an auxiliary lens configured to change the focal length of the optical system (102) is attached, wherein, when it is detected that the auxiliary lens is attached, the processor (101) is configured to perform the process for correcting image distortion caused by aberration of the optical system (102) and the auxiliary lens. [13] The image processing apparatus (100) according to claim 12, wherein, when it is detected that the auxiliary lens is attached, the processor (101) is configured to add a distortion effect based on an aberration caused by the auxiliary lens. [14] The image processing apparatus (100) according to any one of claims 2 to 13, wherein, when a focal length of the optical system (102) is less than a threshold value, the processor (101) is configured to add the distortion effect, and, when the focal length of the optical system (102) is greater than or equal to the threshold value, the processor (101) is configured not to add the distortion effect. [15] The image processing apparatus (100) according to any one of claims 1 to 14, wherein a distortion rate of the second image is smaller than a distortion rate of a distortion caused by aberration of the optical system (102). [16] The image processing apparatus (100) according to any one of claims 1 to 15, wherein the first image is a moving image frame. [17] An image processing device (100) which generates a moving image whose distortion caused by aberration of an optical system (102) used to record the moving image and whose image shake caused by shaking of a device have been corrected, characterized by : Processing the moving image so that a distortion effect is added to each frame of the moving image, using a center position of the frame as a reference position. [18] Image recording device with: the image processing device (100) according to one of claims 1 to 17; and an image sensor (103) configured to generate the first image corresponding to an optical image formed by the optical system (102). [19] Image processing methods with: Creating a second image by applying one or more processes to a first image captured via an optical system (102), wherein the first image has a distortion caused by aberration of the optical system (102), Image center coordinates of the second image and a center of aberration corresponding coordinates do not match, and the second image, which has a point-symmetric distortion centered at the image center coordinates, is generated by applying a transformation process to the first image. [20] Image processing methods with: Generating a moving image whose distortion caused by aberration of an optical system (102) used to record the moving image and whose image blur caused by shaking of a device have been corrected, wherein generating comprises processing the moving image such that a distortion effect is added to each frame of the moving image, using a center position of the frame as a reference position. [21] A program which, when executed by an image processing device (100), causes the image processing device (100) to perform a method according to claim 19 or 20. [22] A computer-readable storage medium storing a program according to claim 21.

Images