Imaging device and control method thereof

Abstract

To provide an imaging apparatus that can arbitrarily control a blurred shape in a captured image by intuitive operation.SOLUTION: An imaging apparatus includes a shift mechanism that shifts a lens with respect to an imaging element, a user shoots subjects 301 and 302 while moving the imaging apparatus within a setting plane (x-y plane). The imaging apparatus measures a distance to a subject and the position of the imaging apparatus, and calculates the amount of lens shift by the shift mechanism on the basis of the measured distance information of the subject and the position information of the imaging apparatus. Images 307 and 308 at different imaging positions are acquired through a lens driven according to the calculated shift amount. Due to addition of the images 307 and 308, the imaging apparatus generates and outputs a composite image 309 in which the blurred shape of the subject 302 is controlled.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present invention relates to the structure and operation of an imaging device for improving the degree of freedom in controlling the image quality of a captured image. [Background technology]

[0002] With the spread of digital cameras and portable communication devices, taking pictures has become widespread. In addition to enjoying pictures for personal use, people are also increasingly using electromagnetic media to share their pictures widely, increasing the opportunities to show pictures to others. This has led to a growing desire for high-quality, unique images.

[0003] Blur is one of the important parameters that indicate the characteristics of an image. One method for adding unique blur is to place an aperture of the desired shape in front of the lens. To change the blur shape with this method, the aperture shape must be changed, and only blur shapes with pre-defined aperture patterns can be realized. Furthermore, when adding blur to an image using image processing, estimation is performed on the image area and amount to apply the blur filter, but processing errors can occur under certain conditions, potentially resulting in an optically unnatural image.

[0004] The imaging device disclosed in Patent Document 1 is equipped with a microlens array as a pupil dividing element that limits the light beam incident on the pixels of the imaging element to a specific pupil area of the photographing lens. By acquiring shape information selected by a user operation and controlling the weighting of pixel data, it is possible to change the blur shape without using a special method when photographing. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Laid-Open No. 2016-220016 Summary of the Invention [Problem to be solved by the invention]

[0006] In the conventional method, there are restrictions on the degree of freedom of image quality control for changing the characteristics of the captured image. For example, assume that the divided area of the pattern related to the shape information is an area of m pixels × n pixels. In this case, the size (number of pixels) of the image becomes 1 / (m × n) of the number of pixels of the original imaging device. That is, if the number of divisions is increased so as to obtain a desired shape, the resolution decreases, and if resolution is emphasized, a detailed shape cannot be specified, which is a trade-off relationship. Since the degree of freedom in shape creation is significantly restricted in a rough pattern, it is difficult for the user to specify a desired blur shape.

[0007] In the method of selecting a blur shape from among the patterns prepared in advance, the user has to select a pattern from limited options. Also, when prior user registration is required with the desired pattern as a block shape, the operation becomes complicated and an intuitive instruction operation is difficult. Further, since the amount of blur that can be applied to the image is restricted by the size of the lens pupil, the range of controllable blur amount is also narrow. An object of the present invention is to provide an imaging device capable of arbitrarily controlling a blur shape in a captured image by an intuitive operation.

Means for Solving the Problem

[0008] The imaging device according to an embodiment of the present invention includes an imaging element that captures a subject through a lens, and moving means capable of moving a subject image on the imaging element, Calculation means for calculating the amount of movement of the subject image control means for performing control to acquire a plurality of images while changing the amount of movement of the subject image by the moving means according to a change in the position of the imaging device, and outputting an image obtained by synthesizing the plurality of images. Well, the control means registers or recognizes a specific subject within the shooting angle of view as the main subject, and the calculation means calculates the amount of movement based on the change in position or speed of the image of the main subject within the shooting angle of view.

Effect of the Invention

[0009] According to the present invention, it is possible to provide an imaging device capable of arbitrarily controlling a blur shape in a captured image by an intuitive operation.

Brief Description of the Drawings

[0010]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Mode for Carrying Out the Invention

[0011] Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the drawings. In the embodiments, a technique that contributes to improving the degree of freedom in image quality control related to captured images will be described in detail. Before describing the embodiments of the present invention, the relationship between the characteristics of the optical system and the amount of blur will be described. Even in a device equipped with an optical system that cannot obtain a large amount of blur in the initial state, if the characteristics of the optical system can be changed by some additional mechanism and a large amount of blur can be obtained, the amount of blur can be arbitrarily selected depending on the operation of the additional mechanism.

[0012] FIG. 1 is an explanatory diagram of the amount of blur and the pupil diameter. FIG. 1(A) shows a lens 103 with a small pupil diameter, and FIG. 1(B) shows a lens 102 with a large pupil diameter. FIG. 1 schematically shows the optical positional relationship among the subject 101, the lens 103 or the lens 102, and the imaging device 104. Light rays from the subject 101, the circle of confusion 105, and the depths of focus 106 and 107 are shown.

[0013] A lens capable of obtaining an image with a large amount of blur is a lens with a small F-number, that is, a lens with a large pupil diameter. The light emitted from the position of the subject 101 shown in FIGS. 1(A) and 1(B) passes through each partial region of the pupil and intersects again at the imaging plane near the imaging device 104. The light rays from the lens 102 with a large pupil diameter have a larger angle when the light transmitted through each partial region of the pupil intersects at the imaging plane on the imaging device 104 compared to the light rays from the lens 103 with a small pupil diameter. When the circle of confusion 105 is kept constant, the depth of focus 106 corresponding to the length within the circle of confusion of the lens 102 with a large pupil diameter is shorter than the depth of focus 107 of the lens 103 with a small pupil diameter. That is, even when the distance from the in-focus subject 101 is the same, the amount of blur becomes larger when the pupil diameter is larger.

[0014] FIG. 2 is a conceptual diagram of an optical system, which will be described by comparing the optical system according to the present invention shown in FIG. 2(A) with the optical system of a comparative example shown in FIG. 2(B). FIG. 2(A) shows a subject 201, lenses 202 and 203 with a small pupil diameter, an image circle 204, and an image sensor 205. The pupil 210 is a virtual pupil including the lenses 202 and 203. FIG. 2(B) shows a subject 201, a lens 206 with a large pupil diameter, an image circle 209, and an image sensor 205. The pupil partial regions 207 and 208 are a part of the pupil of the lens 206.

[0015] The lens 206 with a large pupil diameter shown in FIG. 2(B) can be considered as an aggregate of pupil partial regions (see 207, 208). In this case, it can be said that the images formed on the image sensor 205 by each pupil partial region are optically overlapped.

[0016] On the other hand, the pupil diameters of the lenses 202 and 203 shown in FIG. 2(A) are smaller than that of the lens 206. In this case, an operation is performed to move the camera while maintaining the positional relationship between each lens and the image sensor 205 so that the positions of the subject 201 to be focused and the lenses 202 and 203 are on the same straight line. By superimposing a plurality of images acquired by the camera at this time, it is possible to perform shooting in an optically equivalent relationship to shooting with the virtual pupil 210. That is, an image normally taken with a small-diameter lens 202 or 203 has a small blur because the depth of field of the subject is deep. However, by superimposing a plurality of acquired images, it is possible to obtain an imaging image with a depth of field of the subject equivalent to that in the case of shooting with a large-diameter lens 206.

[0017] Referring to FIG. 3, the process of increasing the amount of blur by overlapping a plurality of images will be described in terms of the angle of view of the image (shooting angle of view). FIG. 3(A) shows the relationship between the positions of different subjects 301, 302, the lens positions 303, 304, and the positions 305, 306 of the imaging device. Subject 301 indicates the subject in focus, and subject 302 is behind subject 301 and indicates the out-of-focus subject. The axis direction perpendicular to the plane of FIG. 3(A) is defined as the y direction, and the directions of two axes orthogonal to each other within the plane of the paper are defined as the x direction and the z direction. The positive direction with respect to the x-axis is the upward direction in FIG. 3, and the z direction is the direction parallel to the optical axis of the imaging optical system.

[0018] In FIG. 3(A), the lens position 303 and the position 305 of the imaging device respectively indicate the lens position and the position of the imaging device when the camera has moved to the right side (the side where the x coordinate value increases) with respect to the subject. Also, the lens position 304 and the position 306 of the imaging device respectively indicate the lens position and the position of the imaging device when the camera has moved to the left side (the side where the x coordinate value decreases) with respect to the subject.

[0019] FIG. 3(B) is a schematic diagram showing an image example. Image 307 shows the image captured at the lens position 303 and the position 305 of the imaging device. Image 308 shows the image captured at the lens position 304 and the position 306 of the imaging device. Image 309 shows the composite image obtained by overlapping image 307 and image 308.

[0020] In the case of the lens position 303 and the position 305 of the imaging device, due to the positional relationship between the subject 301 and the subject 302, the distance between the image of the subject 301 and the image of the subject 302 in the image 307 is relatively large. On the other hand, in the case of the lens position 304 and the position 306 of the imaging device, the distance between the image of the subject 301 and the image of the subject 302 in the image 308 is relatively small. The positional relationship between the lens and the imaging device is determined according to the distance from the camera to the subject 301 to be focused. Therefore, although the position of the subject 301 in the image does not change, the position of the subject 302 in the image changes. That is, although there is no change in the position of the subject 301 in the synthesized image 309, the images of the subject 302 are superimposed as images with different positions, resulting in blurring.

[0021] As can be seen from FIG. 3, by moving the camera in the horizontal direction (x direction), horizontal blurring occurs, and the greater the amount of movement, the greater the amount of blurring. Similarly, by moving the camera in the vertical direction (y direction), vertical blurring occurs, and the greater the amount of movement, the greater the amount of blurring. That is, there is a similar relationship between the movement trajectory of the camera and the shape of the blur. When a user takes a picture using the camera according to an embodiment of the present invention, the user imagines the desired shape of the blur and moves the camera corresponding to that shape, thereby obtaining an image with the blur of that shape added. Therefore, compared with the conventional blur addition method, the degree of freedom of the blur shape can be increased dramatically, and intuitive control based on camera movement can be realized.

[0022] [First Embodiment] Referring to FIG. 4, the configuration of an imaging device for controlling the blur shape of an image will be described. The imaging device 500 is configured such that the lens unit 503 can be moved and rotated by a lens drive unit 502 disposed in a camera housing unit 501. Since the imaging device 500 is moved in various directions within a predetermined plane, the lens unit 503 needs to be configured to be shiftable in various directions. In the plane of FIG. 4, the left-right direction is defined as the x direction, and the up-down direction is defined as the y direction. The direction perpendicular to the plane of FIG. 4 is defined as the z direction, and the z direction is parallel to the optical axis of the imaging optical system of the imaging device 500.

[0023] As an example of an imaging device, there is a device having a lens shift mechanism in the x and y directions. For example, a mechanism for driving a lens in an arbitrary direction within a two-dimensional plane (x-y plane) by an electromagnetic drive unit (electromagnetic coil, magnet) is used. Further, the imaging device 500 shown in FIG. 4 has a structure in which the lens unit 3 can be moved and rotated by a lens drive unit 502 on the camera housing unit 501. With reference to the rotation center axis of the lens drive unit 502, the radial direction is defined as the r direction and the rotation direction is defined as the θ direction.

[0024] The lens unit 503 can be moved in the r direction (radial direction) and rotated in the θ direction by the lens drive unit 502. That is, by moving the lens unit 503 in the r direction and rotating it in the θ direction, the lens unit 503 can be shifted to an arbitrary position. According to this configuration, since it is only necessary to provide a one-dimensional movement mechanism for the lens unit 503 and rotate it, miniaturization and cost reduction of the imaging device can be achieved.

[0025] Since there are limitations on the sizes of the camera housing unit 501 and the lens unit 503, there is a limit to the shift amount. In this embodiment, the shift amount is determined based on the distance (subject distance) information from the imaging device 500 to the subject and the position information of the imaging device 500. When the imaging device moves to a position corresponding to a shift amount equal to or greater than the limit value, the relationship among the subject distance, the position of the imaging device, and the shift amount is disrupted. In this case, there is a possibility that an image not suitable for realizing the intended function of controlling the blur shape is acquired. Therefore, when the shift amount reaches the limit value while the user is moving the imaging device during shooting, it is preferable to take measures to notify the user of this fact. By using a notification device that utilizes sound, light, vibration, etc., it is possible to notify the user that moving the imaging device further will not contribute to the control of the blur shape.

[0026] FIG. 5 is a block diagram showing a configuration example of the imaging device 500. First, the configuration of the camera housing unit 501 will be described. Light from the subject forms an image on the imaging surface of the imaging element 6 through the imaging optical system 3 of the lens unit 503.

[0027] Micro lenses are arranged in a grid pattern on the surface of the imaging element 6. A large number of micro lenses constitute the micro lens array 20. While driving the imaging optical system 3 to move the focal position, signals from the photoelectric conversion units corresponding to the respective micro lenses are acquired, and information related to the distance of the subject (distance information) is obtained by comparing the contrast at each focal position.

[0028] In this embodiment, an example of measuring the subject distance based on the contrast detection method is shown, but distance information of the subject can be obtained by a phase difference detection method on the imaging surface, a method using a phase difference detection sensor separate from the imaging one, a TOF (Time of Flight) method, or the like.

[0029] Also, since signals representing the evaluation amount for focus adjustment and the exposure amount can be obtained from the output of the imaging element 6, AF control and AE (automatic exposure) control of the imaging optical system 3 are possible based on these signals.

[0030] The image processing unit 7 has an A (analog) / D (digital) converter, a white balance circuit, a gamma correction circuit, an interpolation arithmetic circuit, etc. inside, and can generate image data for recording. Also, the image processing unit 7 performs addition processing on a plurality of acquired images, generates a composite image, and outputs it to the memory unit 8.

[0031] The memory unit 8 includes a storage unit and processing circuits necessary for storing image data and the like. The memory unit 8 performs compression processing and decompression processing of data such as images, videos, and sounds by a predetermined method. The image data stored in the memory unit 8 is read out and output to the display unit 9 or the recording and playback unit 10. The display unit 9 includes an LCD (liquid crystal display device) or the like, and performs image display according to a control command from the system control unit 5. Also, the display unit 9 displays a predetermined message on the screen to notify the user.

[0032] The recording and playback unit 10 performs processes of recording image data and the like on a predetermined recording medium or reading and playing back data from the recording medium according to control commands from the system control unit 5. The predetermined recording medium is, for example, a semiconductor memory device that can be mounted on the camera body and used.

[0033] The system control unit 5 includes a CPU (Central Processing Unit) and is the central unit that controls each component of the imaging system. The system control unit 5 generates timing signals and the like during imaging and outputs them to each unit, and also controls each component of the imaging system, image processing system, and recording and playback system in response to operation instruction signals. The system control unit 5 performs processes such as displaying or recording the image data processed by the image processing unit 7, and transmitting to an external device using an output device.

[0034] The operation detection unit 11 detects a user's operation performed using an operation switch, touch panel, etc., and outputs an operation instruction signal to the system control unit 5. For example, the operation detection unit 11 detects a user's shooting operation instruction and notifies the system control unit 5. The system control unit 5 controls the driving of the imaging element 6, the operation of the image processing unit 7, the compression process by the memory unit 8, etc., and controls the display of images and information on the screen of the display unit 9.

[0035] The position and orientation detection unit 12 includes an angular velocity sensor, an acceleration sensor, etc., detects the position and orientation of the imaging device 500, and outputs a detection signal to the system control unit 5. The system control unit 5 acquires the position information, movement amount, and orientation information of the imaging device 500 by a known method based on the detection signal. It is not limited to the method of detecting the position and orientation physically by a sensor device. For example, the imaging device may separately include an imaging unit 21 that detects the movement of the photographer side, and the change in the position of the imaging device may be estimated from the change in the position of the photographer captured by the imaging unit 21. More specifically, a sub-imaging system different from the main imaging system may be provided on the photographer side of the imaging device, and the position of the imaging device 500 can be estimated by the system control unit 5 from the orientation and movement amount of the image of the photographer's body output from the imaging unit 21 including the sub-imaging system. Also, a combination of the estimation method based on the physical information of the sensor device and the estimation method by the sub-imaging system may be used.

[0036] The light emitting unit 13 includes a light source for illuminating a subject, and the light emission amount is controlled by a control signal from the system control unit 5. The system control unit 5 performs control to irradiate illumination light from the light source of the light emitting unit 13 toward the subject as necessary. Details of the light emission control of the light source will be described in the embodiments below.

[0037] Next, the configuration of the lens driving unit 502 will be described. The lens driving unit 502 includes a driving unit 14 and a position detection unit 15. The imaging optical system 3 of the lens unit 503 is composed of optical members such as a lens group and a diaphragm, and the optical axis 4 is shown by a one-dot chain line in FIG. 5.

[0038] The driving unit 14 drives the lens unit 503 in the r direction and the θ direction in FIG. 4 according to a control signal from the system control unit 5. The driving unit 14 performs a shift operation of the lens unit 503 by a driving mechanism unit in the r direction and a driving mechanism unit in the θ direction. Each driving mechanism unit is not limited to a specific configuration, and various mechanisms can be adopted. The position detection unit 15 detects the position (FIG. 4: r, θ) of the lens unit 503 and outputs a position detection signal to the driving unit 14. The driving unit 14 performs feedback driving to reduce the deviation between the position detection signal and the driving target signal.

[0039] Referring to FIG. 6, the operation of the imaging device 500 will be described. The process shown in FIG. 6 is realized by the CPU included in the system control unit 5 executing a predetermined program. First, the composition is determined in S401, and then the origin is registered in S402. For example, the system control unit 5 executes a process of explicitly registering the origin of the imaging device 500 according to a user operation when starting imaging. Alternatively, in the next S403, a process of automatically registering the camera position at the time of measuring the subject distance as the origin is executed.

[0040] In S403, the system control unit 5 performs a subject distance measurement process to obtain distance information from the imaging device 500 to the subject. After the imaging device 500 moves in S404, it proceeds to S405 to measure the movement amount of the imaging device 500. The measurement of the movement amount is performed by the system control unit 5 using the detection signal of the position and attitude detection unit 12.

[0041] In S406, the system control unit 5 calculates the shift amount of the lens unit 503 based on the subject distance and the movement amount of the imaging device. The shift amount corresponds to the movement amount in the r direction and the rotation amount in the θ direction in FIG. 4 in the lens drive unit 502. In S407, the system control unit 5 controls the drive of the lens unit 503 by the drive unit 14 according to the calculated shift amount. Then in S408, the image processing unit 7 acquires the captured image.

[0042] In S409, the system control unit 5 determines whether a predetermined time (threshold time) has elapsed. The predetermined time is set according to conditions such as the desired amount of blur and the speed at which the camera moves. The longer the predetermined time, the longer the time the camera can be moved to control the blur, and it becomes easier to obtain the desired blur shape. On the other hand, the possibility of subject blur increases. Conversely, when the predetermined time is short, subject blur is less likely to occur, but in order to obtain the same amount of blur, the camera needs to be moved faster. If it is determined that the predetermined time has elapsed, the process proceeds to the process of S410. If it is determined that the predetermined time has not elapsed, the process returns to S404 to continue the process.

[0043] In S410, the system control unit 5 executes an addition process on the plurality of images acquired in S408. In the addition process, the signals of the images taken relatively dark may be simply added (the brightness increases by addition), or the signals of the images taken with proper exposure may be averaged by addition. Also, considering the S / N (signal-to-noise) ratio of the images, the signals of the images slightly darker than the proper exposure may be added, and the output may be divided by a number less than the number of added images. In S411, the system control unit 5 performs output (display, recording, external output, etc.) processing on the image signals added in S410, and then ends the series of processes.

[0044] By the way, in order to realize a soft blur shape, there is a lens with an apodization filter built in. This means that for the light transmitted through the peripheral part of the pupil, by reducing the amount of this light, for the light passing through each position of the pupil, the closer to the center, the greater the weighting is optically superimposed.

[0045] FIG. 7 is a schematic diagram showing the relationship between the distance from the origin of the imaging device and the weighting amount of the image for performing such weighting. As different imaging positions, imaging examples at the vicinity of the origin 601 and positions 602 and 603 are shown on the upper side of FIG. 7, and a graph showing the relationship between the imaging position and the weighting amount is shown on the lower side of FIG. 7.

[0046] When superimposing a plurality of images, the weighting amount for the image captured at the vicinity of the origin 601 is relatively large, and the weighting amount for the images captured at positions 602 and 603 away from the origin is relatively small, and addition processing is performed. Thereby, an effect equivalent to that of a configuration incorporating an apodization filter can be obtained. Also, even when the weighting amounts of the plurality of images to be added are equal, by relatively lengthening the time staying near the center when moving the imaging device, an effect equivalent to the above can be obtained.

[0047] Regarding the addition processing of a plurality of images, there are a method in which the image processing unit 7 inside the imaging device adds the captured image signals output from the image sensor 6, and a method in which addition is performed in the memory unit inside the image sensor 6. Also, there is a method of adding on the photoelectric conversion unit (photodiode) constituting the pixel unit of the image sensor 6. In the method of adding the captured image signals in the memory unit in the image sensor, the time required for image processing, recording, and storage can be shortened, so the shooting interval can be narrowed. Also, in the method of adding signals on the photoelectric conversion unit (charge accumulation of electrons or holes), the shooting intervals are continuous, and smoother images can be obtained.

[0048] Predetermined determination conditions can be set for the images to be subjected to the addition processing. For example, when the subject is a person, using face recognition technology, a process of recognizing an image of a subject with temporarily closed eyes and excluding it from the addition targets is performed. A desired image can be obtained by performing the addition processing only on a plurality of images that satisfy the determination conditions.

[0049] While the imaging device is being moved, a shooting operation is performed, and by simply adding the acquired plurality of images, an image controlled to have a desired blur shape can be obtained. However, when a measurement error in the movement amount of the imaging device or an error in the shift operation of the lens unit occurs, a slight blur may occur in the image of the focused subject even if only simple addition is performed. When adding a plurality of images, it is possible to obtain a desired image by recognizing the subject and shifting the images to reduce the deviation before adding them.

[0050] With reference to FIG. 8, the relationship between the lens shift amount and the blur amount will be specifically described. FIG. 8 is a schematic diagram showing the optical positional relationship among a subject 701, a lens 702, and an image sensor 703. For the sake of simplicity of illustration, the optical system is represented as a single lens 702, and the lens barrel is represented as the pupil of the optical system. The pupil 704 is a virtual pupil obtained when the lens 702 is moved. At the origin of the imaging device, the lens 702 is in a state where it is not shifted with respect to the image sensor 703, that is, a state where the optical axis of the lens 702 and the center of the aperture pixel of the image sensor 703 are not deviated.

[0051] Each parameter is defined as follows. · Focal length of the original lens: f · F-number of the original lens: F0 → Aperture diameter of the original lens: A0 = f / F0 · F-number of the virtual pupil: F → Aperture diameter of the virtual lens: A = f / F · Subject distance: d · Required lens shift amount: Smax

[0052] The triangle 705 shown in FIG. 8 represents a triangle with three vertices being the subject 701, the center of the lens 702 at the origin of the imaging device, and the center of the lens 702 at the position of the imaging element 703 after the imaging device has moved. The center of the lens 702 at the origin of the imaging device corresponds to the foot of the perpendicular from the center of the lens 702 to a straight line parallel to the optical axis. Also, the triangle 706 represents a triangle with three vertices being the center of the lens 702, the center of the imaging element 703 before the shift, and the center of the imaging element 703 after the shift. The center of the imaging element 703 before the shift corresponds to the foot of the perpendicular from the center of the imaging element 703 after the shift to a straight line parallel to the optical axis passing through the center of the lens 702.

[0053] Since the triangle 705 and the triangle 706 are in a similar relationship, the following equations (1) and (2) hold.

Equation

[0054] From equation (2), it can be seen that the required lens shift amount Smax increases as the focal length of the original lens is larger, the subject distance is smaller, the F-number of the virtual pupil is smaller (larger aperture), and the F-number of the original lens is larger (smaller aperture). The smaller the aperture of the original lens, the larger the lens shift amount is required to obtain the same virtual pupil. The reason is that the size of the virtual pupil is the sum of the pupil diameter of the original lens and the virtual enlargement of the pupil diameter due to the movement of the imaging device.

[0055] The required movement amount of the camera housing part, that is, the movement amount from the origin of the imaging element 703 when the lens shift amount is the required lens shift amount Smax, is denoted as Sbmax. Since the movement amount Sbmax corresponds to the length obtained by adding the shift amount to the movement amount of the lens center in FIG. 8, the following equation (3) is obtained.

Equation

[0056] As a specific example, when substituting f = 40 mm, F0 = 5.6, F = 1.0, and d = 500 mm into Equation (2), Smax = 1.3 mm. From Equation (3), Sbmax = (40 / 1 - 40 / 5.6) / 2 + 1.3 = 17.7 mm. That is, when a camera equipped with a lens of f = 40 mm and F0 = 5.6 is moved within a range of a circle with a radius of 17.7 mm (the moving range of the lens center is (40 / 1 - 40 / 5.6) / 2 = 16.4 mm), a blur equivalent to F1.0 can be obtained.

[0057] Referring to FIG. 9, the image circle diameter required for the lens will be described. The original image circle is a circle with the diagonal length of the image sensor as the diameter. The image circle after shift is a circle that can cover the area of the image sensor when the lens moves in the positive and negative directions (vertical direction) in FIG. 9 by the maximum shift amount. The definitions of each parameter are the same as those in FIG. 8.

[0058] Let the diameter of the required image circle be denoted as Dim, and the diagonal length of the rectangular image sensor 703 be denoted as w. Dim is obtained from the following Equation (4). [Number]

[0059] When the diagonal length w of the image sensor 703 is 7.7 mm, in the case of f = 40 mm, F0 = 5.6, F = 1.0, and d = 500 mm, from Equation (4), Dim = 1.3×2 + 7.7 = 10.3 mm. The diameter of the required image circle is a 34% increase from the original image circle diameter (7.7 mm). In the above description, the subject distance is defined as the distance from the principal point of the optical system to the subject. The distance from the image sensor to the subject is the length obtained by adding the focal length of the optical system to the above subject distance. In the embodiment of the present invention, the shift amount of the lens is changed based on the changing position of the image sensor and the subject distance. The relationship between the position of the imaging device and the shift amount of the lens is shown in FIG. 10. From the similarity of triangle 1805 and triangle 1806 shown in FIG. 10, there is the following relationship of Equation (5).

Mathematics

Mathematics

[0060] In the above description, as an example of an embodiment to which the present invention is applied, an example of a lens shift type camera in which the imaging element is fixed to the housing of the imaging device and the optical system moves with respect to the housing of the imaging device is shown. Even when the present invention is applied to a sensor shift type camera in which the optical system is fixed to the housing of the imaging device and the imaging element moves, the positional relationship among the subject, the optical system, and the imaging element can be explained equivalently. The arrangement in this case is shown in FIG. 11. From the similarity between triangle 1905 and triangle 1906 shown in FIG. 11, there is the following relationship in Equation (7).

Mathematics

Mathematics

[0061] In this embodiment, a drive unit that shifts a lens or an imaging element according to distance information of a subject and position information of the imaging device is provided, and each image acquired at a plurality of different imaging positions is added and output. According to this embodiment, it is possible to provide an imaging device capable of controlling a desired blur shape by performing an intuitive operation of moving the imaging device by a user.

[0062] [Second Embodiment] Referring to FIG. 12, a second embodiment of the present invention will be described. In this embodiment, a process of acquiring a plurality of images based on movement control of a crop region, which is a difference from the first embodiment, will be described. The crop region is a partial region included in the imaging range, and in crop shooting, a process of extracting an image within the set crop region is performed. Movement control of the crop region corresponds to shift control of the imaging element. Note that for matters similar to those in the first embodiment in this embodiment, the same reference numerals and symbols as those already used are reused, and detailed description thereof is omitted. Such a method of omitting description is the same in the embodiments described later.

[0063] FIG. 12 corresponds to FIG. 2(A) in this embodiment, and is the same as FIG. 2(A) except for the imaging element 905. In this embodiment, instead of the lens shift operation, by moving the crop regions (see 906 and 907) of the imaging element 905 according to the subject distance and the position of the imaging device, an image equivalent to that of the first embodiment can be obtained. In this case, the amount of movement of the crop region corresponds to the amount of shift of the imaging element and can be calculated from the focal length of the lens, the position information of the imaging device, and the distance information of the subject.

[0064] Referring to FIG. 13, the operation of this embodiment will be described. The processes from S1001 to S1005 and from S1008 to S1011 in FIG. 13 are the same as the processes from S401 to S405 and from S408 to S411 in FIG. 6. The processes of S1006 and S1007, which are the differences, will be described.

[0065] Proceeding from S1005 to S1006, the system control unit 5 calculates the crop region of the imaging element 905 from the subject distance and the amount of movement of the imaging device. In S1007, the system control unit 5 performs movement control of the crop region. By this control, it is not necessary to perform shift control of the lens or the imaging element, so the configuration of the imaging device can be simplified.

[0066] FIG. 14 is a flowchart for explaining another operation in this embodiment. The processes from S1101 to S1103 are the same as the processes from S401 to S403 in FIG. 6. After the imaging device moves in S1104, the amount of movement of the imaging device is measured in S1105. Next, an image is acquired in S1106.

[0067] In S1107, the system control unit 5 determines whether or not a predetermined time (threshold time) has elapsed. If it is determined that the predetermined time has elapsed, the process proceeds to S1108. If it is determined that the predetermined time has not elapsed, the process returns to S1104 to continue the process.

[0068] In S1108, the system control unit 5 executes a process of shifting and adding a plurality of acquired images. Next, in S1109, the system control unit 5 sets the overlapping region of the added plurality of images as the crop region for the image processing unit 7, and in S1110, performs a process of outputting the image of the crop region and ends the operation.

[0069] In the operation process of FIG. 14, the calculation of the crop region is not performed at the time of shooting. Imaging is performed in the entire area (effective imaging area) of the imaging element, and crop processing is performed after adding a plurality of images. Regarding the setting position of the crop region, it can be determined not only by the movement amount of the imaging device, but also from the angle of view of the image in the overlapping region of the images when a plurality of images are added so that the positions of the in-focus subjects overlap. By doing so, the error in the position measurement of the imaging device can be absorbed and reduced by the position adjustment when adding the images. Even when shooting a subject in the entire area of the imaging element, the system control unit 5 may show the area to be crop-processed on the screen of the display unit at the time of composition determination, or may display only the area to be cropped. By recognizing the area to be crop-processed on the display screen, the user can more accurately determine the composition.

[0070] According to this embodiment, since a physical shift mechanism unit for the lens or the imaging element is not required, miniaturization and cost reduction of the imaging device can be achieved. Also, while improving the operation followability, an arbitrary blur shape can be obtained.

[0071] [Third Embodiment] Referring to FIGS. 15 and 16, a third embodiment of the present invention will be described. In the first embodiment, according to the relationship of formula (6) or formula (8) depending on the position of the imaging device and the subject distance, the shift amount of the lens or the imaging element is calculated. The shift amount obtained from formula (6) or formula (8) corresponds to the arrangement of the lens and the imaging element that cancels the position change of the subject image on the imaging element when the position of the imaging device changes. In this embodiment, from the information of the captured image itself acquired by the imaging device, the shift amount that cancels the position change of the subject image is calculated, and continuous images are captured while performing the shift operation. The shift amount obtained in this way will result in being the same as the shift amount calculated according to the position of the imaging device and the distance of the subject in the first embodiment.

[0072] FIG. 15 is a schematic diagram showing the relationship of the shooting angle of view in this embodiment. FIG. 15(A) shows the initial state, FIG. 15(B) shows the case without the shift operation, and FIG. 15(C) shows the case with the shift operation. An example of the shooting scene of the main subject 2001 and the background subject 2002 is shown.

[0073] In the initial state shown in FIG. 15(A), within the shooting angle of view 2003, the imaging device is given an instruction by the photographer to register the main subject 2001 to be focused, or the imaging device recognizes the main subject 2001. As a method for recognizing the main subject 2001, various subject detection techniques used for improving the accuracy of autofocus and exposure can be used. In addition, a process of notifying the photographer that the imaging device has recognized the main subject 2001 is performed. For example, a frame 2004 indicating the recognition state of the main subject 2001 is displayed by the display unit 9 in FIG. 5.

[0074] The state of FIG. 15(B) shows the case where the position of the imaging device is moved to the right (the side where the x-coordinate value increases) as viewed from the photographer without the shift operation of the lens or the imaging element. As the imaging device moves, the image of the main subject 2001 moves to the left within the shooting angle of view 2003.

[0075] As shown in the state of FIG. 15(C), in this embodiment, when the position of the image of the main subject 2001 changes as the imaging device moves after the initial state shown in FIG. 15(A), a shift operation of the lens or the imaging element is performed. That is, the shift amount of the lens or the imaging element is calculated from the movement amount of the image of the main subject 2001, and control is performed to shift the lens or the imaging element so as to cancel the position change of the main subject 2001 and obtain the angle of view of FIG. 15(C). Although the position of the imaging device changes between the initial state of FIG. 15(A) and the state of FIG. 15(C), the coordinates of the image of the main subject 2001 on the imaging element do not change. For the images of the background subjects 2002 at different distances from the main subject, the coordinates of the images on the imaging element are different between the initial state of FIG. 15(A) and the state of FIG. 15(C).

[0076] Regarding the method for obtaining the shift amount in this embodiment, there is a method of calculating the difference from the position of the main subject in the previously acquired frame. There is also a method of predicting the future position from the movement speeds of the main subjects in a plurality of frames and performing control so that the position of the main subject within the shooting angle of view is constant. When using the shift of the imaging element, the shift amount of the imaging element for canceling the position change of the main subject is the movement amount of the main subject image itself.

[0077] FIG. 16 is a flowchart for explaining the operation of this embodiment. The composition is determined in S2101. In S2102, registration of the main subject according to the instruction of the photographer or recognition processing of the main subject by the imaging device is performed. Focusing is performed in S2103, and a subject distance measurement process is executed. Note that the processing of S2102 and S2103 may be performed in either order or simultaneously.

[0078] In S2104, the imaging device is moved by the photographer. In S2105, the system control unit 5 measures the movement amount of the main subject image. In S2106, the system control unit 5 calculates the shift amount from the measured movement amount of the main subject image. In S2107, a shift operation of the lens or the imaging element is performed.

[0079] In S2108, the image processing unit 7 acquires the captured image. In S2109, the system control unit 5 determines whether a predetermined time has elapsed. If it is determined in S2109 that the predetermined time has elapsed, the process proceeds to S2110. If it is determined that the predetermined time has not elapsed, the process proceeds to S2104, and the processes from S2104 to S2108 are repeatedly executed until the predetermined time is reached.

[0080] In S2110, the system control unit 5 executes a process of adding a plurality of images. In S2111, the system control unit 5 outputs the signal of the image (added image) added in S2110 and then ends the series of processes.

[0081] In this embodiment, it is not necessary to accurately measure the position of the imaging device or the subject distance. The main subject within the imaging angle of view is registered or recognized, and the shift amount can be calculated based on the change in position or speed of the main subject image within the imaging angle of view. That is, by calculating the shift amount that cancels the change in the position of the subject image from the information of the captured image, the same effect as in the first embodiment can be obtained. Also, in this embodiment, it is not necessary to correct the change in the lens extension amount due to the subject distance, which affects the calculation of the shift amount in the first embodiment, that is, the value of the focal length f or the subject distance d in Equation (6) or Equation (8), according to the subject distance.

[0082] [Fourth Embodiment] With reference to FIGS. 17 to 19, a fourth embodiment of the present invention will be described. FIG. 17 is a schematic diagram showing the configuration of the imaging device 1200 of this embodiment. The direction perpendicular to the plane of FIG. 17 is defined as the y direction, and two directions orthogonal to each other within the plane of the paper are defined as the x direction and the z direction. The z direction is a direction parallel to the optical axis of the imaging optical system of the imaging device 1200.

[0083] The imaging device 1200 includes a lens 1201, a lens barrel 1202, a camera housing portion 1203, springs 1204 and 1205, a weight 1206, an arm 1207, and an imaging element 1209. A part of the arm 1207 is attached to the lens barrel 1202, and a weight 1206 is installed on another part.

[0084] The lens 1201 is integrated with the lens barrel 1202 and is connected to the camera housing part 1203 via springs 1204 and 1205. The springs 1204 and 1205 are elastic members that bias the lens barrel 1202. The lens 1201 is movable in a direction against the elastic forces of the springs 1204 and 1205. The lens barrel 1202 and the weight 1206 are connected via an arm 1207, and the arm 1207 is rotatable about a rotation axis 1208 as a central axis.

[0085] FIG. 18 is an explanatory diagram of a state where a photographer is moving the imaging device 1200 along a circular orbit in the x-y plane. FIG. 18(A) is a view from above of the posture of the photographer taking a picture with the imaging device 1200. The direction orthogonal to the plane of the paper in FIG. 18(A) is the y direction, and the two orthogonal directions within the plane of the paper are the x direction and the z direction. FIG. 18(B) is a view of the photographer and the imaging device 1200 seen from the rear side. The direction orthogonal to the plane of the paper in FIG. 18(B) is the z direction, and the two orthogonal directions within the plane of the paper are the x direction and the y direction. With the center of the circular orbit described by the center of the camera housing part 1203 as a reference (origin), the radial direction is defined as the r direction and the circumferential direction is defined as the θ direction. The state where the camera housing part 1203 is in the position of FIG. 18(B) and the view from above of its interior corresponds to FIG. 17.

[0086] Hereinafter, the operation of the imaging device 1200 will be described assuming a situation where a photographer is moving the camera housing part 1203 along a circular orbit with a radius r at an angular velocity ω. Each parameter is defined as follows. · m1: The mass of the composite system of the lens 1201 and the lens barrel 1202. · m2: The mass of the weight. · l1: The distance from the rotation axis 1208 of the arm 1207 to the plane where the center of gravity of the composite system of the lens 1201 and the lens barrel 1202 can move left and right. · l2: The distance from the rotation axis 1208 of the arm 1207 to the center of gravity of the weight 1206.

[0087] In order to enable a desired shift operation, the structure is set such that the condition shown in the following formula (9) is satisfied.

Equation

Number

[0088] When the imaging device 1200 is moved at a constant angular velocity ω along a circular orbit with a radius r in the x-y plane with the structure of FIG. 17, centrifugal forces F1 and F2 are applied to the combined system of the lens 1201 and the lens barrel 1202 and the weight 1206, respectively. F1 is the centrifugal force applied to the combined system of the lens 1201 and the lens barrel 1202, and F2 is the centrifugal force applied to the weight 1206. The magnitudes of the centrifugal forces F1 and F2 are expressed by the following formulas (10) and (11), taking into account the increase and decrease in the radius due to the shift.

Number

[0089] The combined system of the spring 1204 and the spring 1205 has a spring constant denoted as k, and the force applied to the lens barrel 1202 by these springs is denoted as F3. F3 is expressed by the following formula (12).

Number

[0090] From the balance of the moment about the rotation axis 1208, the following formula (13) holds.

Number

[0091] Substituting Expressions (10), (11), and (12) into Expression (13) gives the following Expression (14).

Equation

[0092] When rearranging with respect to ω, it becomes as follows: the following Expressions (15) and (16).

Equation

[0093] Since r is the radius of rotation corresponding to the position from the center of rotation (origin) of the camera housing unit 1203, it corresponds to Sbmax in Expression (3). Thus, the following Expression (17) is obtained.

Equation

[0094] Since the F-value and focal length of the original lens are known, if the value corresponding to the desired F-value and the distance to the subject are determined, the radius of rotation r can be obtained from Expression (17). That is, by rotating the imaging device 1200 along a circular orbit with this radius of rotation r and the angular velocity ω obtained from Expression (16), a blur corresponding to the desired F-value can be obtained. Note that when the imaging device is moving under the condition of a constant radius of rotation r and angular velocity ω, S corresponds to Smax. However, in the initial state or end state when rotating the imaging device, when r and ω are in a transient state, S may be a value smaller than Smax. Adding the images taken under this condition also fills the central portion of the virtual pupil in the form of a ring generated by the rotational motion, contributing to the realization of a softer blur shape.

[0095] In FIG. 17, for convenience of explanation, a structure within a two-dimensional plane as viewed from the y direction is shown, and thus the weight 1206 and the arm 1207 are depicted in the optical path from the lens 1201 toward the imaging device 1209. Therefore, although it appears that shadows of these members are formed, actually, the members can be three-dimensionally arranged so that no shadow is formed.

[0096] After measuring the subject distance, the system control unit 5 performs a process of displaying the value of the rotation radius r obtained from Equation (17) on the screen of the display unit 9, and notifies the photographer of the range in which the imaging device 1200 moves. Further, as in the example shown in FIG. 19, the system control unit 5 performs a process of displaying the angular velocity ω obtained from Equation (14) in an animation on the screen of the display device 1210 provided on the back surface of the camera housing unit 1203. By viewing the display video, the user can perform more accurate operations.

[0097] According to this embodiment, when the imaging device receives an acceleration, the lens can be shifted by a desired amount by the force. An addition process of a plurality of images captured by the imaging device via the lens is performed, and an added image is output. By omitting the actuator for lens shift, simplification and weight reduction of the configuration can be achieved. Further, it is possible to obtain an arbitrary blur shape while enhancing the responsiveness of the shift with respect to the camera position and reducing power consumption.

[0098] [Fifth Embodiment] Next, a fifth embodiment of the present invention will be described. In an imaging device including a light source that emits auxiliary light to supplement the amount of light under low illuminance conditions, the position of the light source is limited by the housing size. That is, the light source cannot be installed at a position far from the optical axis of the imaging optical system. Further, the area of the light source is also limited by the housing size, so it cannot be made larger than a predetermined value. Therefore, a shadow caused by the light emitted from the light source included in the imaging device may be generated with a sharp edge near the subject, resulting in a very unappealing image. This may also be a cause for the photographer to hesitate to use the auxiliary light.

[0099] In order to move the shadow of the subject away from the subject, the position of the light source may be moved away from the optical axis. In order to smooth the edge of the shadow, the area of the light source may be increased. In this embodiment, in order to control the blur shape, the imaging device is moved within the plane facing the subject (defined as the x-y plane) to perform the imaging operation. By combining the way of moving the imaging device and the way of emitting light from the light source, even when a light source with a small area is installed near the lens, an equivalent light source shape can be freely changed.

[0100] Referring to FIG. 20, the imaging operation in this embodiment will be described. FIGS. 20(a) to 20(d) are explanatory diagrams of the position of the camera with respect to the origin in the x-y plane and the light emission pattern of the light source included in the light emitting unit 13. FIGS. 20(A) to 20(D) are explanatory diagrams of the shadow of the subject obtained with the light source arrangement corresponding to each of FIGS. 20(a) to 20(d). The white star marks represent the state in which the light source is emitting light, and the black star marks represent the state in which the light source is turned off.

[0101] When the imaging device is located at the origin, it is assumed that the position of the optical axis is also approximately at the origin. FIG. 20(a) shows the light emission position of the light source when the imaging device including the light source is at the origin. The light source irradiates light at a position offset by a distance Δr from the optical axis to the installation location of the light source on the camera housing. In this arrangement, the distance from the optical axis to the light source is short and the area of the light source is also small. Therefore, as shown in FIG. 20(A), an unsightly shadow 1511 is generated on the subject.

[0102] When using the imaging device of this embodiment, the photographer moves the imaging device within the range of the circle 1501 shown hatched in FIG. 20(b). The moving range of the light source is the range shown by the circle 1502. As shown in FIG. 20(b), if the light source is made to emit light only on the outermost circle 1502 of the moving range of the imaging device, the subject is in a state as if it is irradiated by a ring-shaped light source. Therefore, as shown in FIG. 20(B), almost no shadow is generated on the subject.

[0103] FIG. 20(c) shows a case where the light source emits light at some positions where the imaging device has moved significantly from the origin. In this case, since the subject is in a state as if it is irradiated by a point light source away from the origin of the imaging device, a shadow 1512 away from the subject is generated as shown in FIG. 20(C). Further, FIG. 20(d) shows a case where the light source emits light at a position when the imaging device moves within a specific region (for example, the first quadrant of the x-y plane). In this case, since the subject is in a state as if it is irradiated by a surface light source away from the origin of the imaging device, a smooth shadow 1513 of the edge away from the subject is generated as shown in FIG. 20(D).

[0104] In order to control the shape of the shadow of the subject to a desired shape, there are the following methods. · A method of pre-selecting a light emission pattern according to the position from the origin and controlling the brightness of the light source when the imaging device comes to that position. · A method of registering in advance how much light the light source emits at which position by performing an input operation with a button or the like while the user moves the imaging device, and controlling the brightness of the light source when the imaging device comes to the registered position.

[0105] FIG. 21 is a flowchart for explaining the operation process of this embodiment. First, in S1601, the system control unit 5 executes a registration process of the light emission pattern of the light source. After the composition is determined in the next S1602, in S1603, the system control unit 5 executes a registration process of the origin. The process of explicitly registering the origin by a user operation may be performed, or it may be a process in which the camera position at the time of the subject distance measurement operation in S1604 is automatically registered as the origin.

[0106] Next, in S1604, the system control unit 5 measures the subject distance. After the imaging device moves in S1605, in S1606, the system control unit 5 measures the amount of movement of the imaging device. In S1607, the system control unit 5 calculates the lens shift amount based on the subject distance and the amount of movement of the imaging device. In S1608, the system control unit 5 controls the shift operation of the lens according to the calculated lens shift amount. In S1609, the system control unit 5 calculates the light emission amount of the light source from the registered light emission pattern and the amount of movement of the imaging device. The light source emits light in S1610 according to the calculated light emission amount. Regarding the processes of S1609 and S1610, the order may be swapped with S1607 and S1608, or they may be performed simultaneously.

[0107] Next to S1610, an image is acquired in S1611. In S1612, the system control unit 5 determines whether a predetermined time has elapsed. If it is determined that the predetermined time has elapsed, the process proceeds to S1613. If it is determined that the predetermined time has not elapsed, the process returns to S1605 to continue the processing. In S1613, the system control unit 5 executes an addition process on the acquired multiple images, performs an output process on the added image in S1614, and then ends the series of processes.

[0108] In the measurement of the amount of movement of the imaging device in S1606, it may be performed using a resolution equivalent to the resolution when determining the lens shift amount. Alternatively, the resolution of the measurement of the amount of movement used for light source control may be different from the resolution of the measurement of the amount of movement when determining the lens shift amount. Also, the amount of movement of the imaging device may be obtained from the relationship between the amount of movement on the imaging plane of the registered main subject image and the focal length.

[0109] According to this embodiment, an imaging device having a light source for illuminating a subject can be moved within the x - y plane to freely control not only the blur shape but also the shape of the shadow generated on the subject.

[0110] According to each of the above embodiments, an imaging image with an arbitrary blur shape can be obtained by a camera movement operation performed by the user, which contributes to an improvement in the degree of freedom of image quality control. As described above, the preferred embodiments of the present invention have been explained, but the present invention is not limited to the above embodiments, and combinations of the configurations shown in each embodiment, as well as various modifications and changes, are possible.

Explanation of Reference Numerals

[0111] 104, 205, 305, 306, 703, 905, 1209: Image sensor 303, 304, 503, 702, 1201: Lens 501, 1203: Camera housing part 502: Lens drive unit 1202: Lens barrel 1204, 1205: Spring 1206: Weight 1207: Arm 2001: Main subject

Claims

1. An imaging device that images a subject through a lens, Moving means capable of moving a subject image on the imaging device, Calculating means for calculating the amount of movement of the subject image, Control means for acquiring a plurality of images while changing the amount of movement of the subject image by the moving means in accordance with a change in the position of the imaging device, and performing control to output an image obtained by synthesizing the plurality of images, The control means registers or recognizes a specific subject within the shooting angle of view as the main subject, The calculating means calculates the amount of movement based on a change in position or speed of the image of the main subject within the shooting angle of view. An imaging device characterized by the above.

2. The moving means is a driving means for driving the lens, The lens is shifted with respect to the imaging device by the moving means. The imaging device according to claim 1, characterized by the above.

3. The moving means is a driving means for driving the imaging device, The imaging device is shifted with respect to the lens by the driving means. The imaging device according to claim 1, characterized by the above.

4. First acquisition means for acquiring distance information to the subject, Second acquisition means for acquiring position information of the imaging device, The calculating means calculates the amount of movement of the subject image from the distance information and the position information. The imaging device according to any one of claims 1 to 3, characterized by the above.

5. The calculating means calculates the amount of movement so that the subject, the principal point of the lens, and the region for acquiring the image of the imaging device are in a positional relationship on the same straight line. The imaging device according to claim 4, characterized by the above.

6. First acquisition means for acquiring distance information to the subject, Second acquisition means for acquiring position information of the imaging device, When calculating the shift amount for moving the lens, the calculating means calculates by dividing the product of the focal length of the lens and the position information by the sum of the distance information and the focal length of the lens. The imaging device according to claim 2, characterized by the above.

7. First acquisition means for acquiring distance information to the subject, Second acquisition means for acquiring position information of the imaging device, When calculating the shift amount for moving the imaging device, the calculating means calculates by dividing the product of the focal length of the lens and the position information by the distance information. The imaging device according to claim 3, characterized by the above.

8. The control means performs a process of registering the origin of the position information, and calculates the movement amount from the position information based on the registered origin and the distance information. The imaging device according to claim 4 or claim 5, characterized in that.

9. The control means performs a process of registering the origin of the position information, and calculates a region for acquiring the image from the position information based on the registered origin and the distance information. The imaging device according to claim 7, characterized in that.

10. The control means performs a process of registering, as the origin, the position of the imaging device when the distance information is acquired, or the position of the imaging device when imaging is started. The imaging device according to claim 8 or claim 9, characterized in that.

11. The control means performs control to output an image obtained by adding the plurality of images in a memory unit included in the imaging element, or an image obtained by adding in a photoelectric conversion unit constituting a pixel unit of the imaging element. The imaging device according to any one of claims 1 to 10, characterized in that.

12. The moving means performs driving to move the lens or the imaging element in a first direction and driving to rotate the lens or the imaging element in a second direction orthogonal to the first direction. The imaging device according to claim 2 or claim 3, characterized in that.

13. The control means performs a process of notifying that the shift of the lens or the imaging element has reached the limit. The imaging device according to claim 2 or claim 3, characterized in that.

14. The control means performs control to output an image obtained by adding the plurality of images with weighting corresponding to the distance of the imaging device from the origin. The imaging device according to any one of claims 8 to 10, characterized in that.

15. The control means performs control to output an image obtained by synthesizing the plurality of images that satisfy the set conditions. The imaging device according to any one of claims 1 to 14, characterized in that.

16. The control means performs control to cancel the movement of the image by moving the imaging element with respect to the vibration received by the imaging device. The imaging device according to claim 2, characterized in that.

17. The control means performs control to cancel the movement of the image by moving the lens constituting the imaging optical system with respect to the vibration received by the imaging device. The imaging device according to claim 3, characterized in that.

18. A control method executed by an imaging device including an imaging element that images a subject, A step of imaging a subject by the imaging device; A calculating step of calculating the amount of movement of the subject image by a calculating means; A step of moving the subject image on the imaging device; A control step of acquiring a plurality of images while changing the amount of movement of the subject image according to the position change of the imaging device and outputting an image obtained by synthesizing the plurality of images, which is performed by a control means; In the control step, the control means registers or recognizes a specific subject within the imaging angle of view as a main subject, and in the calculating step, the calculating means calculates the amount of movement based on the change in position or speed of the image of the main subject within the imaging angle of view; A control method of an imaging device, characterized in that.

19. A control method executed by an imaging device including an imaging element that images a subject through a lens and a driving means that shifts the lens with respect to the imaging element or shifts the imaging element with respect to the lens, A first acquisition step of acquiring distance information to the subject; A second acquisition step of acquiring position information of the imaging device; A calculating step of calculating the shift amount of the lens or the imaging element for each imaging position from the distance information and the position information; And a step of performing control to output an image obtained by synthesizing a plurality of images captured by the imaging element at different imaging positions by controlling the driving means with the shift amount. A control method of an imaging device, characterized in that.

20. A control method executed by an imaging device including an imaging element that images a subject through a lens and a driving means that shifts the lens with respect to the imaging element or shifts the imaging element with respect to the lens, A step of registering or recognizing a specific subject within the imaging angle of view as a main subject; An acquisition step of acquiring the amount of movement of the image of the main subject on the imaging plane; A calculating step of calculating the shift amount of the lens or the imaging element for each imaging position; And a step of performing control to output an image obtained by synthesizing a plurality of images captured by the imaging element at different imaging positions by controlling the driving means with the shift amount. A control method of an imaging device, characterized in that.

21. A control method executed by an imaging device including an imaging element that images a subject, A step of imaging a subject through a lens by the imaging element; A calculating step of calculating the amount of movement of the subject image by a calculating means; A step of moving the subject image on the imaging device using a mechanism unit that shifts the imaging element; A step of performing image blur correction using an optical system that shifts the lens; A control step in which control means performs control to acquire a plurality of images while changing the movement amount of the subject image according to a change in the position of the imaging device, and outputs an image obtained by synthesizing the plurality of images; In the control step, the control means registers or recognizes a specific subject within the shooting angle of view as a main subject, and in the calculation step, the calculation means calculates the movement amount based on a change in position or speed of the image of the main subject within the shooting angle of view; A control method for an imaging device, characterized by the above.

22. A control method executed by an imaging device including an imaging element that images a subject, the method comprising: A step of imaging a subject through a lens by the imaging element; A calculation step in which a calculation means calculates a movement amount of the subject image; A step of moving the subject image on the imaging element using an optical system that shifts the lens; A step of performing image blur correction using a mechanism unit that shifts the imaging element; A control step in which control means performs control to acquire a plurality of images while changing the movement amount of the subject image according to a change in the position of the imaging device, and outputs an image obtained by synthesizing the plurality of images; In the control step, the control means registers or recognizes a specific subject within the shooting angle of view as a main subject, and in the calculation step, the calculation means calculates the movement amount based on a change in position or speed of the image of the main subject within the shooting angle of view; A control method for an imaging device, characterized by the above.

Images