Image processing device and method, electronic device, program, and storage medium

Abstract

To make a change in a displayed image less noticeable when a mode change accompanies a change in the size and / or position of an image cutout range.SOLUTION: An image processing device includes input means for inputting an image, processing means for cutting out a partial image in a predetermined range from the image, control means for controlling the size and the position of the range, and switching means for switching between a plurality of modes involving the process of cutting out the partial image, and when the size and the position of the range change as a result of the switching of the mode by the switching means, the control means immediately changes the size and the position of the range, and when either the size or the position of the range changes, the control means gradually changes whichever of the size and the position of the range is changed.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates to an image processing device and method, an electronic device, a program, and a storage medium, and to a technique for stabilizing a displayed image. [Background technology]

[0002] Conventionally, there have been imaging devices equipped with image stabilization devices that stabilize moving images caused by camera shake. These imaging devices reduce image shake using so-called optical or electronic shake correction. Optical shake correction drives an image stabilization unit, such as a shake correction lens or an image sensor, to counteract camera shake in response to camera shake information detected by a camera shake detection unit. Electronic shake correction uses image processing to crop a small area of ​​the captured image relative to the imaging area in order to counteract camera shake.

[0003] Furthermore, a subject tracking technology is also generally known in which a subject is detected from a captured image and the detected subject is tracked and kept at a predetermined position within the angle of view by controlling the cropping position of image processing in electronic image stabilization. With this technology, when subject tracking is started or ended, it becomes necessary to shift the position of the image cropping range (for example, the center position of the cropping range) between a predetermined position (for example, the center position) of the captured image and the target position for subject tracking.

[0004] Patent Document 1 proposes a method in which, when the cropped angle of view is changed at the start or end of electronic shake correction, the displayed image is gradually shifted from a first angle of view to a second angle of view at a predetermined rate over multiple frames, thereby reducing the sense of discomfort that accompanies the change in angle of view at the start or end of electronic shake correction. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Patent No. 4793639 Summary of the Invention [Problem to be solved by the invention]

[0006] However, even if the image cutout range is gradually shifted as described in Patent Document 1, if the size of the image cutout range changes depending on whether or not subject tracking is being performed, there is a risk that a new sense of incongruity will appear on the display screen when subject tracking begins or ends.

[0007] The present invention has been made in consideration of the above problems, and aims to make changes in the displayed image less noticeable when changing modes that involve changes in the size and / or position of the image cropping range. [Means for solving the problem]

[0008] In order to achieve the above object, the image processing device of the present invention has an input means for inputting an image, a processing means for cutting out a partial image of a predetermined range from the image, a control means for controlling the size and position of the range, and a switching means for switching between a plurality of modes involved in the process of cutting out the partial image, wherein the control means performs a first control in which, when the size and position of the range change due to the switching of the mode by the switching means, the control means immediately changes the size and position of the range, and when either the size or the position of the range changes, the control means gradually changes whichever of the size and position of the range is changed. [Effects of the Invention]

[0009] According to the present invention, when a mode change involves a change in the size and / or position of the image cropping range, the change in the displayed image can be made less noticeable. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a block diagram showing an example of the configuration of an imaging apparatus according to an embodiment of the present invention. [Figure 2] FIG. 2 is a block diagram showing an example of the configuration related to image blur correction control and subject tracking control in the first embodiment. [Figure 3] 4 is a flowchart showing control in the first embodiment. [Figure 4] 4 is a time chart showing an example of a tracking control amount in the first embodiment. [Figure 5] 6 is a time chart illustrating a problem in the second embodiment. [Figure 6] 10 is a flowchart showing control in the second embodiment. [Figure 7] 10 is a time chart showing an example of a tracking control amount in the second embodiment. [Figure 8] 10 is a time chart showing an example of a tracking control amount and a size of a cutout range in the third embodiment. [Figure 9] 10 is a time chart showing the problem that occurs when switching from subject tracking mode to electronic shake correction mode. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention claimed. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted. First Embodiment FIG. 1 is a block diagram showing an example of the configuration of an imaging device according to this embodiment. In this embodiment, a digital camera will be described as an example of an imaging device. However, the present invention is not limited to this, and may be applied to various electronic devices equipped with a camera function. For example, the imaging device according to the present invention may be a mobile communication terminal with a camera function such as a mobile phone or a smartphone, a mobile computer with a camera function, a mobile game console with a camera function, or the like.

[0012] Furthermore, the imaging device of this embodiment is configured to include a camera body 1 and a lens unit 2 that is detachably attached to the camera body 1, but it may also be configured as an integrated unit.

[0013] The zoom lens 101 optically changes the focal length of the imaging optical system (imaging lens) 200 by moving in the optical axis direction, thereby changing the imaging angle of view. The vibration-proof lens 102 optically corrects image blur caused by vibration of the imaging device by moving in the direction perpendicular to the optical axis. The focus lens 103 optically adjusts the focus position by moving in the optical axis direction. The iris 104 and shutter 105 can adjust the amount of light by opening and closing, and are used for exposure control.

[0014] Light passing through the photographing optical system 200 is received by an image sensor 106 using a CCD, CMOS sensor, etc. via a shutter 105 configured in the camera body 1, and is converted into an electrical signal (image signal) corresponding to the amount of light by photoelectric conversion. An AD converter 107 performs noise removal processing, gain adjustment processing, and AD conversion processing on the image signal read out from the image sensor 106, and outputs a digital image signal. The timing generator 108 controls the drive timing of the image sensor 106 and the output timing of the AD converter 107 in accordance with instructions from the camera control unit 115 .

[0015] The image processing circuit 109 performs processes such as pixel interpolation and color conversion on the digital image signal output from the AD converter 107, and sends the processed digital image signal (image data) to an internal memory 110. The image processing circuit 109 also includes a circuit for aligning multiple images captured in succession, a geometric transformation circuit for performing cylindrical coordinate transformation and lens group distortion correction, and a compositing circuit for performing trimming and compositing processes. Electronic shake correction is performed using a projective transformation circuit provided in the image processing circuit 109. Note that the operation of each circuit is well known, so a detailed description will be omitted.

[0016] The internal memory 110 stores image data, shooting information, various control programs, and the like. The display unit 111 displays image data stored in the internal memory 110, shooting information, and the like. The compression / decompression processing unit 112 performs compression or decompression processing on the image data stored in the internal memory 110 according to the image format. The storage memory 113 stores various data such as parameters. The operation unit 114 is a user interface that allows the user to perform various menu operations, mode switching operations, and the like.

[0017] The camera control unit 115 is composed of a computing device such as a CPU (Central Processing Unit), and executes various control programs stored in the internal memory 110 in response to user operations via the operation unit 114. The control programs include, for example, programs for performing zoom control, image stabilization control, automatic exposure control, automatic focus adjustment control, and processing for detecting the face of a subject. The camera control unit 115 will be described in detail below, and processing in each unit is realized by the CPU executing the control programs stored in the internal memory 110.

[0018] The luminance signal detector 137 detects the luminance signal of the subject or the entire image based on the digital image signal read from the image sensor 106 and converted by the AD converter 107, and calculates the luminance.

[0019] The exposure control unit 136 calculates an exposure value (aperture value and shutter speed) based on the luminance obtained by the luminance signal detection unit 137, and notifies the diaphragm driving unit 120 and the shutter driving unit 135 of the calculation result. The exposure control unit 136 also determines a gain value used in gain adjustment processing in the AD converter 107, and notifies the AD converter 107 of the gain value. This performs automatic exposure control (AE control). In the case of a lens-interchangeable camera, information is transmitted between the camera body 1 and the lens unit 2 via the camera side communication unit 140 and the lens side communication unit 128 . The aperture driver 120 and the shutter driver 135 drive the aperture 104 and the shutter 105, respectively.

[0020] The evaluation value calculation unit 138 extracts a specific frequency component from the luminance signal obtained by the luminance signal detection unit 137, then calculates a contrast evaluation value based on the extracted luminance signal of the specific frequency, and outputs it to the focus lens control unit 139.

[0021] Focus lens control unit 139 issues a command to drive focus lens 103 by a predetermined drive amount over a predetermined range, while acquiring a contrast evaluation value at each focus lens position from evaluation value calculation unit 138. As a result, the defocus amount in the contrast AF method is calculated from the focus lens position where the curve of change in the contrast evaluation value reaches its peak, and notifies focus lens driving unit 121 of this amount via camera side communication unit 140 and lens side communication unit 128. The focus lens driving unit 121 drives the focus lens 103 based on the notified defocus amount, thereby performing automatic focusing control (AF control) to focus the light beam on the light receiving surface of the image sensor 106. Although the contrast AF method has been described here, a phase difference AF method may also be used, and since the phase difference AF method is well known, a description thereof will be omitted.

[0022] The zoom lens control unit 127 determines the drive amount of the zoom lens 101 in accordance with a zoom operation instruction from the operation unit 114, and the zoom lens drive unit 124 changes the angle of view by driving the zoom lens 101 based on the drive amount. The determined drive amount is also output to the lens vibration isolation control unit 126.

[0023] Camera-side shake detection unit 134 is, for example, a gyro sensor, and detects shake or vibration applied to the imaging device. In this embodiment, in addition to camera-side shake detection unit 134 arranged on the camera body 1 side, lens-side shake detection unit 125, which is, for example, configured by a gyro sensor, is also arranged on the lens unit 2 side to detect shake or vibration applied to the lens. The motion vector detection unit 131 uses a block matching method to calculate the correlation value between the current frame image and the previous frame image for each of the multiple blocks into which the frame image is divided, then searches for the block in the previous frame image that has the smallest correlation value, and detects the deviation of other blocks relative to that block as the motion vector between the images.

[0024] The camera stabilization control unit 133 on the camera body 1 side can communicate with the lens stabilization control unit 126 on the lens unit 2 side via the camera communication unit 140 and the lens communication unit 128 in the photographing optical system 200. The camera stabilization control unit 133 on the camera body 1 side calculates an image sensor shake correction amount for suppressing shake using the image sensor 106 based on shake detection signals detected by the camera shake detection unit 134, the lens shake detection unit 125, or both. Then, based on the calculated shake correction amount and the position of the image sensor 106 detected by the image sensor position detection unit 132, it transmits a drive signal to the image sensor drive unit 130 to move the position of the image sensor 106. Based on the image sensor drive signal received from the camera stabilization control unit 133, the image sensor drive unit 130 drives the image sensor 106 in a direction perpendicular to the optical axis.

[0025] The vibration-proof lens position detector 123 detects the position of the vibration-proof lens 102 . Lens vibration reduction control unit 126 calculates a shake correction amount to suppress shake in response to a shake detection signal detected by lens-side shake detection unit 125, camera-side shake detection unit 134, or both. Then, based on the calculated shake correction amount and the position of vibration reduction lens 102 detected by vibration reduction lens position detection unit 123, it transmits a drive signal to vibration reduction lens drive unit 122 to move the position of vibration reduction lens 102. Based on the drive signal received from lens vibration reduction control unit 126, vibration reduction lens drive unit 122 drives vibration reduction lens 102 in a direction perpendicular to the optical axis.

[0026] The subject detection unit 141 generates subject detection information by detecting the image area of ​​the subject contained in the image based on the digital image signal output from the AD converter 107. The subject detection information includes information such as the type of subject (e.g., person, animal, vehicle), part (e.g., pupil, face, body), position, size, etc.

[0027] The subject identification unit 143 sets a specific subject in the captured image. The photographer can set any subject as the tracking target subject by performing a touch operation or button operation using the operation unit 114. Note that the tracking target subject may be determined by an automatic subject setting program of the camera, even if the photographer does not operate the operation unit 114. The subject tracking calculation unit 142 calculates the amount of tracking of the subject, and details will be described later with reference to FIG.

[0028] FIG. 2 is a block diagram showing an example of the configuration related to image blur correction control and subject tracking control described in the first embodiment. First, we will describe the camera vibration isolation control unit 133 on the side of the camera body 1. Note that, to simplify the explanation, the explanation will be given assuming that the vibration detection signal detected by the lens-side vibration detection unit 125 is not used.

[0029] The camera-side integrator 1331 converts the angular velocity of the shake detected by the camera-side shake detector 134 into a shake angle by integrating the angular velocity of the shake. Here, the camera-side integrator 1331 uses an integral low-pass filter (LPF).

[0030] The shake correction amount calculation unit 1332 calculates the shake correction amount that cancels the shake angle, taking into consideration the frequency band of the shake angle and the drivable range of the image sensor 106. Specifically, the shake correction amount is calculated by multiplying the shake angle by gains related to the zoom magnification and the subject distance.

[0031] The division ratio calculation unit 1333 calculates the correction ratio to be performed by the camera body 1 when the sum of the shake correction amounts on the camera body 1 side and the lens unit 2 side is 100%. In this embodiment, the correction ratio is determined based on the respective movable ranges of the image sensor 106 and the vibration-proof lens 102. In addition to the movable ranges of these correction members, the correction ratio may also be determined taking into consideration the movable range for correction by clipping in image processing (electronic vibration prevention). A correction ratio accumulator 1334 multiplies the calculation result from the division ratio calculator 1333 by the shake correction amount, and calculates the final shake correction amount based on the correction ratio.

[0032] The position control unit 1335 performs PID control (ratio control, integral control, and fine control) on the deviation between the target position determined by the shake correction amount of the image sensor 106 and the current position, converts it into an image sensor drive signal, and outputs it to the image sensor drive unit 130. The current position is the output result of the image sensor position detection unit 132. Since PID control is a common technique, a detailed description will be omitted. The image sensor drive unit 130 drives the image sensor 106 in accordance with the image sensor drive signal.

[0033] The electronic shake correction setting unit 1337 and the electronic shake correction amount calculation unit 1336 realize electronic shake correction (electronic vibration reduction), that is, image blur correction by cutting out a small area from the imaging area using image processing and controlling the cut-out position depending on the direction and magnitude of the image blur.

[0034] The electronic shake correction setting unit 1337 accepts settings related to electronic shake correction from the user via the operation unit 114. In this embodiment, multiple electronic shake correction modes are provided depending on the level of effect, and the size of the image cut-out range (for example, 60%, 80%, etc. of the entire imaging area) differs depending on the mode. Here, it is possible to select whether or not to perform electronic shake correction, and the strength of the correction effect using multiple electronic shake correction modes, and the size of the cut-out range from the imaging area is determined depending on the selection.

[0035] The electronic shake correction amount calculation unit 1336 calculates the amount of electronic shake correction corresponding to the shake detection on the camera body 1 side, which has been calculated by the division ratio calculation unit 1333, in accordance with the size of the cut-out range calculated by the electronic shake correction setting unit 1337. Using the amount of electronic shake correction calculated by the electronic shake correction amount calculation unit 1336 as input, the image processing circuit 109 performs electronic shake correction by cutting out an image of the set cut-out range at predetermined coordinates from the captured image.

[0036] Next, we will explain the lens vibration isolation control unit 126 on the lens unit 2 side. Note that, to simplify the explanation, the explanation will be given assuming that the shake detection signal detected by the camera-side shake detection unit 134 is not used.

[0037] The lens-side integrator 1261 converts the angular velocity of the shake detected by the lens-side shake detector 125 into a shake angle by integrating the angular velocity of the shake. Here, the lens-side integrator 1261 uses an integral LPF.

[0038] The shake correction amount calculation unit 1262 calculates the shake correction amount that cancels the shake angle, taking into consideration the frequency band of the shake angle and the driveable range of the vibration-proof lens 102. Specifically, the shake correction amount is calculated by multiplying the shake angle by gains related to the zoom magnification and the subject distance.

[0039] The correction ratio accumulator 1263 calculates the amount of shake correction based on the correction ratio by multiplying the total amount of shake correction on the camera body 1 side and the lens unit 2 side by the correction ratio provided by the lens unit 2 side, assuming that the total amount of shake correction on the camera body 1 side and the lens unit 2 side is 100%. In this embodiment, the correction ratio provided by the lens unit 2 side is calculated from the result of calculation by the division ratio calculator 1333 configured on the camera body 1 side. The correction ratio provided by the lens unit 2 side is notified via the communication units of the camera side communication unit 140 and the lens side communication unit 128.

[0040] The position control unit 1264 performs PID control (ratio control, integral control, fine control) on the deviation between the target position determined by the shake correction amount of the vibration-proof lens 102 and the current position, converts it into a lens drive signal, and outputs it to the vibration-proof lens drive unit 122. The current position is the output result of the vibration-proof lens position detection unit 123. Since PID control is a common technique, a detailed explanation will be omitted. The vibration-proof lens drive unit 122 drives the vibration-proof lens 102 in accordance with the lens drive signal.

[0041] As described above, by driving the vibration-proof lens 102 and the image sensor 106 and changing the image cutout range, image blur caused by camera shake can be reduced.

[0042] Next, the subject tracking calculation unit 142 on the camera body 1 side will be described. In this embodiment, as described above, the subject tracking calculation unit 142 can change the position of the image cropping range based on the subject detection information acquired from the subject detection unit 141. The subject identification unit 143 can set any subject in the captured image as the main subject. The subject detection unit 141 obtains information such as position information, size, and type of subject related to the any subject set by the subject identification unit 143.

[0043] The subject target position calculation unit 1424 receives a target position on the image where the subject is to be held. The target position may be, for example, the center of the screen, a position on the screen touched by the user by operating the operation unit 114, or a coordinate position stored in advance. In this embodiment, for simplicity of explanation, the center of the screen is set as the subject target position.

[0044] The tracking control amount calculation unit 1423 calculates a tracking control amount according to the target position set by the subject target position calculation unit 1424 and the position of the subject detected by the subject detection unit 141. The tracking control amount calculated by the tracking control amount calculation unit 1423 is input to the image processing circuit 109, which performs image processing, in this embodiment, geometric transformation processing similar to electronic shake correction (electronic vibration reduction). In this manner, subject tracking processing is performed.

[0045] The cropped image size change determination unit 1425 determines whether or not a change has occurred in the size of the cropped range cropped from the captured image by the image processing circuit 109, based on various user settings set on the operation unit 114.

[0046] The display mode determination unit 1426 determines, based on various user settings set on the operation unit 114, whether the display unit 111 is in a state where a captured image is being displayed, or in a menu display state where only setting menu information is being displayed, as described below.

[0047] Next, the control procedure in this embodiment will be described with reference to the flowchart in Fig. 3. Here, the mode change between object tracking mode and electronic shake correction mode will be described. Even if the image clipping range for electronic shake correction is gradually changed, if the size of the image clipping range changes depending on whether object tracking is being performed, there is a risk that a new sense of incongruity will appear on the display screen when object tracking starts or ends. This issue will be described with reference to Fig. 9.

[0048] FIG. 9 is a schematic timing chart showing the time-series changes in the position of the image crop area when switching from the object tracking mode to the electronic image stabilization mode at time t0. During the object tracking mode, the position of the image crop area indicated by the solid line 1001 (e.g., the center position of the image crop area) moves to track the tracking target position indicated by the dashed line 1002, thereby keeping the object at a predetermined position in the cropped image (e.g., the center position of the image). When switching from the object tracking mode to the electronic image stabilization mode, which has a different image crop size, at time t0, the tracking control amount gradually transitions to 0 (center position) from time t0 to t1 to avoid a sudden change in the position of the image crop area. However, in this example, the size of the image crop area also changes from 60% to 80% of the imaging area at time t0. Generally, the size of the image crop area often changes quickly. As a result, the appearance of the display screen is such that, after a sudden change in the size of the image crop area at time t0, the position of the image crop area continues to move gradually toward the center of the imaging area from time t0 to t1. The combination of these two changes creates a significant sense of incongruity. By switching between the subject tracking mode and the electronic stabilization mode shown in Figure 3, changes in the displayed image can be made less noticeable when switching modes that involve changes in the size and / or position of the image crop area.

[0049] When a user's mode change request is received from the operation unit 114, the cropped image size change determination unit 1425 determines in S301 whether the size of the cropped area will change before and after the change. Here, the presence or absence of an image size change is determined by comparing the size of the cropped area in the subject tracking mode with the size of the cropped area for shake correction set in the electronic shake correction setting unit 1337. If it is determined that the size of the cropped area will not be changed, the process proceeds to S302, where the tracking control amount calculation unit 1423 controls the tracking control amount so that the position of the cropped area gradually transitions to the new position over a predetermined time. On the other hand, if it is determined that the size of the cropped area will be changed, the process proceeds to S303, where the tracking control amount is controlled so that the position of the cropped area transitions immediately.

[0050] The control in S302 and S303 will be specifically described below. FIG. 4 is a time chart showing a specific example of the change in the tracking control amount over time when transitioning from the subject tracking mode to the electronic shake correction mode, in which the solid lines 401 and 403 represent the tracking control amount and the dashed lines 402 and 404 represent the tracking target position.

[0051] FIG. 4(a) shows an example in which the size of the crop area changes before and after a mode change. At time t0, the camera is switched from an object tracking mode in which the crop area size is 60% to an electronic image stabilization mode in which the crop area size is 80%. In the resulting object tracking mode, tracking control is achieved to keep the detected object at a target position in the cropped partial image so that it is positioned at tracking target position 402, as indicated by tracking control amount 401. When the object tracking mode ends at time t0, the tracking control amount is immediately changed to 0, and the position of the crop area is shifted to its initial position (center) simultaneously with the change in size of the crop area. In this way, by simultaneously and immediately shifting the size and position of the crop area, it is possible to reduce the sense of discomfort felt when the size and position of the crop area change stepwise at different times, as described above with reference to FIG. 9.

[0052] 4(b) shows an example in which the size of the cutout area remains the same before and after the mode change. At time t0, the camera switches from the object tracking mode, in which the size of the cutout area is 60%, to the electronic image stabilization mode, in which the size of the cutout area is also 60%. As in FIG. 4(a), when the object tracking mode ends at time t0, the tracking control amount is gradually shifted to 0 toward time t1, and the position of the cutout area is gradually shifted to its initial position (center). In this way, when the size of the cutout area remains the same, the gradual shifting of the position of the cutout area can alleviate the sense of discomfort felt by the user when viewing the display screen.

[0053] 4, the case where the mode is changed from the object tracking mode to the electronic shake correction mode is described, but the present invention is not limited to this and may be applied, for example, when the mode is changed from the electronic shake correction mode to the object control mode. The size of the cropping range is also an example and is not limited to this.

[0054] Furthermore, the present invention can also be applied to cases where, for example, in the subject tracking mode and electronic shake correction mode, modes in which the size and position of the cutout range differ depending on the strength of the subject tracking or shake correction can be selected. In such cases, when only the size of the cutout range changes, the size may be changed gradually instead of the position. In other words, whether the change is immediate or gradual can be controlled depending on whether the size or position of the cutout range is changed.

[0055] As described above, according to the first embodiment, when a mode change involves a change in the size and / or position of the cropping range of an image, the change in the displayed image can be made less noticeable.

[0056] <Second embodiment> Next, a second embodiment of the present invention will be described. In the first embodiment, a case was described in which the amount of tracking control is controlled according to the size of the cropping range when switching between the object tracking mode and the electronic shake correction mode. In contrast, in the second embodiment, a case will be described in which the amount of tracking control is also controlled according to the display mode when the user instructs switching between the object tracking mode and the electronic shake correction mode. Note that the configuration of the imaging device in the second embodiment is the same as that described in the first embodiment with reference to FIGS. 1 and 2, and therefore a description thereof will be omitted here.

[0057] In this embodiment, there are two display modes: a live view (LV) display in which captured images are always displayed on the display unit 111, and a state in which the LV display is interrupted and only setting menu information is displayed on the display unit (hereinafter referred to as "menu display"). Here, if the same control as in the first embodiment is performed when switching from subject tracking mode to electronic shake correction mode during menu display, for example, a problem different from that in the first embodiment may arise. This problem will be described using FIG. 5.

[0058] 5, like FIG. 4(b), shows an example of a time-series change in the tracking control amount, with a solid line 501 representing the tracking control amount and a dashed line 502 representing the tracking target position. Here, a case is shown in which the display screen is in a menu display state when a mode change is made at time t0. Here, if the cropping range sizes in the subject tracking mode and the electronic shake correction mode are the same (e.g., 60%), the cropping range size does not change. Therefore, if the tracking control amount is gradually transitioned as described in the first embodiment, the transition in the image cropping position remains from the time of return to LV display until time t1, which may cause the LV display to look unnatural.

[0059] Therefore, in the second embodiment, when the mode is changed while the menu is displayed, the tracking control amount is immediately changed to 0 regardless of whether the size of the cropping range has been changed.

[0060] FIG. 6 is a flowchart illustrating control in the second embodiment. When a user's mode change request is received from the operation unit 114, in S601 the display mode determination unit 1426 determines whether or not a menu is being displayed on the display unit 111. If it is determined that a menu is being displayed, the process proceeds to S403, where the tracking control amount is immediately transitioned regardless of whether or not the size of the cropping range has been changed.

[0061] On the other hand, if it is determined in S601 that the display unit 111 is not displaying a menu, in S401 to S403, the tracking control amount is controlled based on whether or not the size of the cropping range has been changed, as in the first embodiment described above.

[0062] 7 is a time chart showing changes in the tracking control amount when it is determined that the display unit 111 is in a menu display state in the second embodiment. This shows an example in which, at time t0, the mode is changed from an object tracking mode with a 60% crop area to an electronic shake correction mode with a 60% crop area. A solid line 701 represents the tracking control amount, and a dashed line 702 represents the tracking target position. As shown in FIG. 7, the tracking control amount is immediately transitioned to 0 at the same time as the mode is changed from the object tracking mode to the electronic shake correction mode at time t0.

[0063] This prevents the screen transition from remaining when returning from menu display to LV display. Also, because the menu is being displayed, the position of the cropped image does not immediately transition, which creates a strange visual effect.

[0064] In the above example, the amount of tracking control is controlled depending on whether the display unit 111 is in a menu display state or not. However, this is not limited to menu display, and when LV display is not being performed, control may be performed to instantly transition the position of the cut-out image.

[0065] As described above, according to the second embodiment, changes in the displayed image can be made less noticeable.

[0066] <Third embodiment> Next, a third embodiment of the present invention will be described. In the first and second embodiments, when the size of the crop area is changed, the tracking control amount is always immediately changed, but some users may feel uncomfortable with this sudden change in the LV display. Therefore, in this embodiment, both the image size and the tracking control amount are changed gradually. Note that the configuration of the imaging device in the third embodiment is the same as that described in the first embodiment with reference to Figures 1 and 2, so a description thereof will be omitted here.

[0067] The control in this embodiment will be described below with reference to FIG.

[0068] 8, solid line 801 represents the tracking control amount, dashed line 802 represents the tracking target position, and solid line 803 represents the size of the crop area. At time t0, when a user issues an instruction to change from electronic shake correction mode with a crop area size of 80% to subject tracking mode with a crop area size of 60%, in this embodiment, both the crop area size and tracking control amount gradually transition from time t0 to t1. By gradually transitioning both the size and position of the crop area in this way, it is possible to avoid the awkward feeling that would occur if the screen were to change suddenly.

[0069] However, in the case of the above-mentioned control, it is preferable to perform processing such that the tracking control amount transitions gradually so that the cut-out partial image does not deviate from the imaging range, for example, by setting a limiter for the tracking control amount as the image size changes.

[0070] While FIG. 8 illustrates an example in which the size of the crop area is reduced by changing the settings, this is not limiting. Similar control may also be used when the crop area size is increased. Alternatively, different control may be used when the crop area size is reduced and when it is increased. For example, when the crop area size is increased, the position of the crop area is changed more abruptly (the change is made in a shorter time) than when the crop area size is reduced. When the crop area size is increased so that the image becomes wider, the change in the position of the crop area is less noticeable than when the crop area size is reduced. Therefore, even when the position of the crop area is changed abruptly, the crop area can be quickly changed to the desired position without creating a sense of incongruity. Alternatively, when the crop area size is reduced, the position of the crop area is changed more abruptly than when the crop area size is increased. When the crop area size is reduced so that the image becomes more enlarged, the position intended by the photographer is more likely to deviate from the crop area than when the crop area size is increased. Therefore, by making the position of the cut-out range steep, it is possible to make it difficult for the position intended by the photographer to fall outside the cut-out range.

[0071] As described above, according to the third embodiment, when a mode change involves a change in the size and / or position of the cropping range of an image, the change in the displayed image can be made less noticeable.

[0072] The first and third embodiments are common in that, when the size of the cutout range changes before and after a mode change, the position of the cutout range is changed in a shorter time than when the size of the cutout range does not change before and after the mode change. For example, assume that a mode change to electronic image stabilization occurs while tracking a subject performing a predetermined specific movement in subject tracking mode. Here, the position of the cutout range before the mode change is defined as a first position, and the target position of the cutout range after the mode change is defined as a second position. The time required to change the position of the cutout range from the first position to the second position is varied depending on whether the size of the cutout range changes before and after the mode change. More specifically, when the size of the cutout range changes, the time required to change the position of the cutout range from the first position to the second position is shorter than when the size of the cutout range does not change. Reducing the time required to change the position of the cutout range from the first position to the second position corresponds to increasing the amount of change in the position of the cutout range between consecutive frames.

[0073] <Other embodiments> The present invention may be applied to a system made up of a plurality of devices, or to an apparatus made up of a single device.

[0074] The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0075] <Summary> The disclosure of this embodiment includes the following configuration.

[0076] (Item 1) an input means for inputting an image; a processing means for cutting out a partial image of a predetermined range from the image; control means for controlling the size and position of said area; a switching means for switching between a plurality of modes that involve the process of cutting out the partial image, The image processing device is characterized in that the control means performs a first control in which, when the size and position of the range change due to the switching of the mode by the switching means, the control means immediately changes the size and position of the range, and, when either the size or the position of the range changes, the control means gradually changes whichever of the size and position of the range changes. (Item 2) further comprising a selection means for selecting one of a plurality of display modes including a first display mode in which the partial image is displayed on the display means and a second display mode in which the partial image is not displayed; The control means When the first display mode is selected, the first control is performed; When the second display mode is selected, a second control is performed to immediately change the size and position of the range, regardless of whether the size and position of the range change as a result of the mode switching by the switching means. 2. The image processing device according to item 1, (Item 3) an input means for inputting an image; A processing means for cutting out a partial image of a predetermined range from the image; control means for controlling the size and position of said area; a switching means for switching between a plurality of modes that involve the process of cutting out the partial image, The image processing device is characterized in that the control means controls the size and position of the range to gradually change when at least one of the size and position of the range changes due to the mode switching by the switching means. (Item 4) an input means for inputting an image; a processing means for cutting out a partial image of a predetermined range from the image; control means for controlling the size and position of said area; a switching means for switching between a plurality of modes that involve the process of cutting out the partial image, The image processing device is characterized in that the control means changes the position of the range in a shorter time when the size of the range changes before and after the mode change due to the mode switching by the switching means than when the size of the range does not change before and after the mode change. (Item 5) an acquisition means for acquiring the amount of shake from the detection means; a detection means for detecting a predetermined subject from the image, the plurality of modes include a first mode in which the position of the range is moved based on the amount of shake, and a second mode in which the position of the range is moved based on the position of the subject, The control means controls the size and position of the range when switching between the first mode and the second mode. 5. The image processing device according to any one of items 1 to 4, characterized in that: (Item 6) 6. The image processing device according to item 5, wherein the first mode is a mode for performing image stabilization, and the second mode is a mode for tracking a subject. (Item 7) 7. The image processing device according to item 5 or 6, further comprising a setting means for setting the subject. (Item 8) 8. The image processing device according to any one of items 5 to 7, wherein the detection means detects the amount of shake based on a motion vector between the images inputted consecutively. (Item 9) 9. The image processing device according to any one of items 5 to 8, wherein the detection means includes a gyro sensor. (Item 10) An image processing device according to any one of items 1 to 9, An imaging means for capturing an image; An electronic device comprising: (Item 11) An image processing method for cutting out a partial image of a predetermined range from an input image, comprising: a determination step of determining whether a switching between a plurality of modes involving a process of cutting out the partial image has been performed; a first change step of immediately changing the size and position of the range when it is determined in the determination step that the mode has been switched and the size and position of the range have changed due to the mode switching; a second change step of gradually changing the size or position of the range when it is determined in the determination step that the mode has been switched and either the size or the position of the range changes due to the mode switching; An image processing method comprising: (Item 12) a selection step of selecting one of a plurality of display modes including a first display mode in which the partial image is displayed on a display means and a second display mode in which the partial image is not displayed; a third change step of immediately changing the size and position of the range when the second display mode is selected and it is determined in the determination step that the mode has been switched, regardless of whether the size and position of the range change as a result of the mode switching; When the first display mode is selected, the first change step or the second change step is performed. Item 12. The image processing method according to item 11. (Item 13) An image processing method for cutting out a partial image of a predetermined range from an input image, comprising: a determination step of determining whether a switching between a plurality of modes involving a process of cutting out the partial image has been performed; a changing step of gradually changing the size and position of the range when it is determined in the determining step that the mode has been switched and at least one of the size and position of the range changes due to the mode switching; An image processing method comprising: (Item 14) An image processing method for cutting out a partial image of a predetermined range from an input image, comprising: a determination step of determining whether a switching between a plurality of modes involving a process of cutting out the partial image has been performed; a changing step of changing the position of the range in a shorter time when it is determined in the determining step that the mode has been switched and the size of the range changes before and after the mode change due to the mode switching than when the size of the range does not change before and after the mode change; An image processing method comprising: (Item 15) A program for causing a computer to function as each of the means of the image processing device according to any one of items 1 to 9. (Item 16) Item 16. A computer-readable storage medium storing the program described in item 15.

[0077] The invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0078] 1: camera body, 2: lens unit, 102: vibration-proof lens, 106: image sensor, 109: image processing circuit, 110: internal memory, 111: display unit, 114: operation unit, 115: camera control unit, 122: vibration-proof lens driving unit, 123: vibration-proof lens position detection unit, 125: lens side shake detection unit, 126: lens vibration prevention control unit, 130: image sensor driving unit, 131: motion vector detection unit, 132: image sensor position detection unit, 133: camera vibration prevention control unit, 134: camera side shake detection unit, 141: subject detection unit, 142: subject tracking calculation unit, 143: subject identification unit

Claims

1. an input means for inputting an image; a processing means for cutting out a partial image of a predetermined range from the image; control means for controlling the size and position of said area; a switching means for switching between a plurality of modes that involve the process of cutting out the partial image, The image processing device is characterized in that the control means performs a first control in which, when the size and position of the range change due to the switching of the mode by the switching means, the control means immediately changes the size and position of the range, and when either the size or the position of the range changes, the control means gradually changes whichever of the size and position of the range changes.

2. The display device further includes a selection unit for selecting one of a plurality of display modes including a first display mode in which the partial image is displayed on the display unit and a second display mode in which the partial image is not displayed, The control means When the first display mode is selected, the first control is performed; When the second display mode is selected, a second control is performed to immediately change the size and position of the range, regardless of whether the size and position of the range change as a result of the mode switching by the switching means.

2. The image processing device according to claim 1, wherein:

3. an input means for inputting an image; a processing means for cutting out a partial image of a predetermined range from the image; control means for controlling the size and position of said area; a switching means for switching between a plurality of modes that involve the process of cutting out the partial image, The image processing device is characterized in that the control means controls the size and position of the range to gradually change when at least one of the size and position of the range changes due to the mode switching by the switching means.

4. an input means for inputting an image; a processing means for cutting out a partial image of a predetermined range from the image; control means for controlling the size and position of said area; a switching means for switching between a plurality of modes that involve the process of cutting out the partial image, The image processing device is characterized in that the control means changes the position of the range in a shorter time when the size of the range changes before and after the mode change due to the mode switching by the switching means than when the size of the range does not change before and after the mode change.

5. an acquisition means for acquiring the amount of shake from the detection means; a detection means for detecting a predetermined subject from the image, the plurality of modes include a first mode in which the position of the range is moved based on the amount of shake, and a second mode in which the position of the range is moved based on the position of the subject, The control means controls the size and position of the range when switching between the first mode and the second mode.

5. The image processing device according to claim 1, wherein the image processing device is a computer.

6. 6. The image processing apparatus according to claim 5, wherein the first mode is a mode for performing image stabilization, and the second mode is a mode for tracking an object.

7. 6. The image processing apparatus according to claim 5, further comprising setting means for setting the subject.

8. 6. The image processing apparatus according to claim 5, wherein said detecting means detects the amount of shake based on a motion vector between said images inputted successively.

9. 6. The image processing apparatus according to claim 5, wherein the detecting means includes a gyro sensor.

10. An image processing device according to any one of claims 1 to 4; An imaging means for capturing an image; An electronic device comprising:

11. An image processing method for cutting out a partial image of a predetermined range from an input image, comprising: a determination step of determining whether a switching between a plurality of modes involving a process of cutting out the partial image has been performed; a first change step of immediately changing the size and position of the range when it is determined in the determination step that the mode has been switched and the size and position of the range have changed due to the mode switching; a second changing step of gradually changing the size or the position of the range when it is determined in the determining step that the mode has been switched and either the size or the position of the range changes due to the mode switching; An image processing method comprising:

12. a selection step of selecting one of a plurality of display modes including a first display mode in which the partial image is displayed on a display means and a second display mode in which the partial image is not displayed; a third change step of immediately changing the size and position of the range when the second display mode is selected and it is determined in the determination step that the mode has been switched, regardless of whether the size and position of the range change as a result of the mode switching; When the first display mode is selected, the first change step or the second change step is performed.

12. The image processing method according to claim 11.

13. An image processing method for cutting out a partial image of a predetermined range from an input image, comprising: a determination step of determining whether a switching between a plurality of modes involving a process of cutting out the partial image has been performed; a changing step of gradually changing the size and position of the range when it is determined in the determining step that the mode has been switched and at least one of the size and position of the range changes due to the mode switching; An image processing method comprising:

14. An image processing method for cutting out a partial image of a predetermined range from an input image, comprising: a determination step of determining whether a switching between a plurality of modes involving a process of cutting out the partial image has been performed; a changing step of changing the position of the range in a shorter time when it is determined in the determining step that the mode has been switched and the size of the range changes before and after the mode change due to the mode change than when the size of the range does not change before and after the mode change; An image processing method comprising:

15. A program for causing a computer to function as each of the means of the image processing apparatus according to any one of claims 1 to 4.

16. A computer-readable storage medium storing the program according to claim 15.

Images