Image capture device and method for controlling image capture device

Abstract

To provide an imaging apparatus for acquiring image data whose resolution is increased by pixel shift and a control method thereof which suppress an information amount of image data subjected to synthesis processing by reducing the number of image data to be acquired by the pixel shift.SOLUTION: An imaging apparatus comprises: an imaging lens; an imaging element in a lamination structure; a pixel shift mechanism; and a pixel shift synthesis processing unit. The pixel shift synthesis processing unit comprises: an image data input part which inputs two pieces of image data acquired by pixel shift; an image data arrangement part which arranges image data in each layer constituting each of the two pieces of image data by shifting the image data; a re-sampling part which re-generates the image data in each layer by re-sampling processing from the image data in each layer arranged by being shifted; a pixel mixing part which generates one piece of image data by pixel mixing processing by using the re-generated image data in each layer; and an image data output part which outputs one piece of image data generated by the pixel mixing processing.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an imaging device that acquires image data by an image sensor and a control method thereof, and more particularly to an imaging device that employs an image sensor having a stacked structure in which photoelectric conversion units are stacked, and acquires image data with a higher resolution by pixel shifting, and a control method thereof. [Background technology]

[0002] Conventionally, imaging devices that acquire image data using an imaging element having a color filter have been widely used. In each pixel constituting the imaging element having a color filter, a color filter of either red (R), green (G) or blue (B) is disposed in front of a photodiode, which is a photoelectric conversion unit. This allows the pixel values ​​of the wavelength components corresponding to the color filters disposed in each pixel to be acquired by the photodiode. Then, the array of pixel values ​​acquired in each pixel is acquired as image data, and image data for recording is acquired by performing demosaic processing on this image data.

[0003] There is also known an imaging device that acquires image data with high resolution by pixel shifting. In pixel shifting, multiple pieces of image data are acquired by shifting all or part of the image sensor or the photographing lens on a plane perpendicular to the optical axis. Then, the acquired multiple pieces of image data are synthesized to finally acquire high-resolution image data.

[0004] Patent Document 1 discloses an electronic still camera that acquires image data by synthesizing data of four images acquired by pixel shifting. It also discloses an electronic still camera that can acquire image data of higher quality by synthesizing data of eight images acquired by pixel shifting.

[0005] Patent Document 2 discloses an imaging device that acquires image data by synthesizing data of eight images acquired by pixel shifting, and that can reduce the capacity of a buffer memory by performing chromatic aberration correction before synthesizing the image data acquired by pixel shifting.

[0006] In addition, there is known an imaging element that can directly obtain, at each pixel, a pixel value corresponding to the wavelength component of the subject light that has been vertically color-separated by stacking multiple photodiodes. Hereinafter, such an imaging element may be referred to as an imaging element having a stacked structure.

[0007] Patent document 3 discloses an imaging element with a stacked structure (Vertical color filter sensor group array), and further discloses an imaging element in which the top layer has full resolution and lower layers have lower resolution in order to eliminate the complexity of the wiring structure of the imaging element with a stacked structure. [Prior art documents] [Patent documents]

[0008] [Patent Document 1] Japanese Patent Application Publication No. 6-225317 [Patent Document 2] JP 2017-46186 A [Patent Document 3] U.S. Patent No. 7,339,216 Summary of the Invention [Problem to be solved by the invention]

[0009] However, the conventional techniques disclosed in the above-mentioned patent documents etc. have the following problems.

[0010] In an image sensor having a color filter, each pixel can only obtain one pixel value corresponding to the color filter. Therefore, in order to obtain image data with a high resolution by pixel shifting using an image sensor having a color filter that employs a general Bayer type array pattern, eight pieces of image data must be obtained by pixel shifting. If the number of pieces of image data to be obtained by pixel shifting increases, the time required to obtain the image data and the time required to perform correction processing, synthesis processing, etc. on the image data will become longer. In addition, the capacity of the buffer memory required to record multiple pieces of image data will become larger.

[0011] When image data less than eight images are acquired by pixel shifting using an imaging element having color filters and the high-resolution image data is then synthesized, the color filter arrangement of the high-resolution image data cannot be made the same as that of normal image data, and interpolation processing, replacement processing, etc. are required to compensate for this. Furthermore, in this case, it becomes difficult to acquire high-quality high-resolution image data that is finally acquired by the synthesis processing.

[0012] On the other hand, a stacked image sensor can directly obtain pixel values ​​corresponding to the wavelength components of the subject light at each pixel. Therefore, when acquiring image data with a high resolution by pixel shifting using a stacked image sensor, the number of image data to be acquired by pixel shifting can be reduced.

[0013] However, since each layer of the image data acquired by the imaging element with a stacked structure has image data with the same amount of information for each wavelength component of the subject light, the amount of information of the image data acquired by the imaging element with a stacked structure becomes very large. When the amount of information of the image data acquired by the pixel shifting using the imaging element with a stacked structure becomes large, the burden of transferring the image data and the burden of performing the synthesis process become large. [Means for solving the problem]

[0014] In order to solve the above problems, an imaging device according to a first aspect of the present invention is an imaging device that generates image data with a higher resolution than normal image data by synthesis processing using a plurality of image data pieces acquired by pixel shifting, and includes a photographing lens that focuses subject light onto an imaging surface as a subject image, an imaging element having a stacked structure in which photoelectric conversion units that convert the subject image into an image signal are stacked, a pixel shifting mechanism that shifts the relative position between the subject image and the imaging element of the stacked structure on a plane perpendicular to an optical axis, and a pixel shift synthesis processing unit that generates one image data piece by synthesis processing from the plurality of image data pieces acquired by pixel shifting, and the pixel shift synthesis processing unit includes an image data input unit that inputs two image data pieces acquired by pixel shifting, an image data arrangement unit that arranges image data of each layer constituting each of the two image data pieces by shifting, and a pixel shift synthesis processing unit that synthesizes one image data piece from the image data of each layer arranged by shifting. Using image data from the same layer The image processing apparatus is characterized by comprising a resampling unit that regenerates image data for each layer through a resampling process, a pixel mixing unit that generates a single piece of image data through a pixel mixing process using the regenerated image data for each layer, and an image data output unit that outputs the single piece of image data generated through the pixel mixing process.

[0015] In addition, the imaging device according to the second invention is characterized in that the pixel shifting mechanism shifts the relative position of the subject image and the stacked imaging element in a plane perpendicular to the optical axis by moving the stacked imaging element in a direction perpendicular to the optical axis.

[0016] In addition, the imaging device according to a third aspect of the present invention is characterized in that the pixel shifting mechanism shifts the relative position of the subject image and the stacked imaging element in a plane perpendicular to the optical axis by moving all or a part of the taking lens in a direction perpendicular to the optical axis.

[0017] In addition, the imaging device according to a fourth aspect of the present invention is characterized in that the pixel shifting mechanism shifts the relative position of the subject image and the stacked imaging element by 1 / 2 the pixel pitch of the stacked imaging element in the XY directions on a plane perpendicular to the optical axis.

[0018] In addition, the imaging device of the fifth invention is characterized in that the pixel shift synthesis processing unit separates the image data of each layer that constitutes two pieces of image data acquired by pixel shifting using the stacked imaging element into a luminance layer and a chrominance layer, and by a resampling process, regenerates image data in the luminance layer that has a higher resolution than normal image data, and regenerates image data in the chrominance layer that has a lower resolution than normal image data.

[0019] Further, an imaging device according to a sixth aspect of the present invention comprises: An imaging device that generates image data with a higher resolution than normal image data by synthesis processing using a plurality of image data acquired by pixel shifting, the imaging device comprising: a photographing lens that focuses subject light onto an imaging surface as a subject image; an imaging element having a laminated structure in which photoelectric conversion units that convert the subject image into an image signal are laminated; a pixel shifting mechanism that shifts the relative position between the subject image and the imaging element having the laminated structure on a plane perpendicular to an optical axis; and a pixel shift synthesis processing unit that generates one image data by synthesis processing from the plurality of image data acquired by pixel shifting, The pixel shift synthesis processing unit The image data output unit is provided with an image data input unit for inputting two pieces of image data acquired by pixel shifting, an image data arrangement unit for arranging the image data of each layer constituting each of the two pieces of image data with a shift, a resampling unit for regenerating image data of each layer from the image data of each layer arranged with a shift by a resampling process, a pixel mixing unit for generating one piece of image data by a pixel mixing process using the regenerated image data of each layer, and an image data output unit for outputting one piece of image data generated by the pixel mixing process, wherein the image data of each layer constituting each of the two pieces of image data acquired by the pixel shifting is divided into a luminance layer and a chrominance layer, and image data having a higher resolution than normal image data is regenerated by the resampling process in the luminance layer, and image data having a lower resolution than normal image data is regenerated in the chrominance layer, The image data of the layer with the highest contrast among the image data of each layer constituting each of the two image data acquired by pixel shifting is set as the luminance layer, and the image data of the other layers is set as the color layers.

[0020] Moreover, an imaging device according to a seventh aspect of the present invention comprises: An imaging device that generates image data with a higher resolution than normal image data by synthesis processing using a plurality of image data acquired by pixel shifting, the imaging device comprising: a photographing lens that focuses subject light onto an imaging surface as a subject image; an imaging element having a laminated structure in which photoelectric conversion units that convert the subject image into an image signal are laminated; a pixel shifting mechanism that shifts the relative position between the subject image and the imaging element having the laminated structure on a plane perpendicular to an optical axis; and a pixel shift synthesis processing unit that generates one image data by synthesis processing from the plurality of image data acquired by pixel shifting, The pixel shift synthesis processing unit The image data output unit is provided with an image data input unit for inputting two pieces of image data acquired by pixel shifting, an image data arrangement unit for arranging the image data of each layer constituting each of the two pieces of image data with a shift, a resampling unit for regenerating image data of each layer from the image data of each layer arranged with a shift by a resampling process, a pixel mixing unit for generating one piece of image data by a pixel mixing process using the regenerated image data of each layer, and an image data output unit for outputting one piece of image data generated by the pixel mixing process, wherein the image data of each layer constituting each of the two pieces of image data acquired by the pixel shifting is divided into a luminance layer and a chrominance layer, and image data having a higher resolution than normal image data is regenerated by the resampling process in the luminance layer, and image data having a lower resolution than normal image data is regenerated in the chrominance layer, The image data of the layers constituting each of the two image data acquired by pixel shifting is characterized in that the image data of the layer with the highest signal value is designated as the luminance layer, and the image data of the other layers are designated as color layers.

[0021] Further, a control method of an imaging device according to an eighth aspect of the present invention is a control method of an imaging device that generates image data having a higher resolution than normal image data by a synthesis process using a plurality of image data acquired by pixel shifting, the control method including an image data acquisition step of acquiring two image data by pixel shifting using an imaging element having a stacked structure, an image data input step of inputting the acquired two image data, an image data arrangement step of arranging the image data of each layer constituting each of the input two image data by shifting them, and an image data arrangement step of arranging the image data of each layer by shifting them by resampling process. Image data from the same layerThe method is characterized by having a resampling processing step of regenerating image data for each layer using the resampled data, a pixel mixing processing step of generating one piece of image data using the image data for each layer regenerated by the pixel mixing processing, and an image data output step of outputting the generated piece of image data. Effect of the Invention

[0022] According to the present invention, in an imaging device and a control method for an imaging device that uses multiple pieces of image data acquired by pixel shifting to generate image data with a higher resolution than normal image data through a synthesis process, the number of pieces of image data to be acquired by pixel shifting can be reduced by using an imaging element with a stacked structure in which photoelectric conversion units that convert a subject image into an image signal are stacked.

[0023] In addition, by dividing the image data of each layer constituting two pieces of image data acquired by pixel shifting using a stacked imaging element into a luminance layer and a color layer, and regenerating image data in the luminance layer with a higher resolution than normal image data by a resampling process, and regenerating image data in the color layer with a lower resolution than normal image data, even in an imaging device and a control method for an imaging device that acquires multiple pieces of image data by pixel shifting using a stacked imaging element, it is possible to reduce the amount of information in the image data to be synthesized and to prevent an increase in the burden on image data transfer and image processing. [Brief description of the drawings]

[0024] [Figure 1] 1 is a system block diagram showing an imaging apparatus according to an embodiment of the present invention. [Diagram 2] FIG. 2 is a schematic diagram showing the depth-wise structure of an imaging element having a three-layer structure. [Diagram 3] FIG. 13 is a schematic diagram showing a method of arranging two pieces of image data acquired by pixel shifting. [Figure 4] FIG. 2 is a schematic diagram showing a state in which image data of each layer is shifted and arranged based on a pixel pitch P. [Diagram 5]FIG. 13 is a schematic diagram illustrating a method of resampling a luminance layer. [Figure 6] FIG. 13 is a schematic diagram illustrating a method of resampling processing of a color layer. [Figure 7] 4 is a flowchart showing steps of a photographing operation of the imaging apparatus of the present embodiment. [Figure 8] 4 is a flowchart showing steps of acquiring image data by pixel shifting. [Figure 9] 11 is a flowchart showing steps of a pixel shift synthesis process. [Figure 10] 4 is a flowchart showing steps of a pixel mixing process. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0025] Hereinafter, an embodiment of an imaging device and a control method for an imaging device according to the present invention will be described in detail with reference to the drawings. In the description of the embodiment, components necessary for explaining the technical features of the present invention will be listed and described.

[0026] (Figure 1: Components of an imaging device) The components of the imaging device of this embodiment will be described with reference to Fig. 1. Fig. 1 is a system block diagram showing the imaging device of this embodiment. In Fig. 1, 100 denotes the imaging device, 110 denotes a photographing lens, 120 denotes an imaging element, 130 denotes a shift mechanism, 140 denotes a pixel shift synthesis processing unit, 150 denotes a bus, 160 denotes a DRAM, and 170 denotes an external memory.

[0027] (Figure 1: Components of an imaging device) The photographing lens 110 is an optical system that converts subject light into a subject image on an imaging plane. The image sensor 120 converts the subject image into an image signal (analog signal) using photodiodes, which are photoelectric conversion units, arranged two-dimensionally on the imaging plane, and acquires the image. The imaging device 100 of this embodiment employs an image sensor with a stacked structure for the image sensor 120. The image sensor with a stacked structure will be described later.

[0028] The shift mechanism 130 moves the image sensor 120 in a direction perpendicular to the optical axis. In the image sensor 100 of this embodiment, the shift mechanism 130 functions as a pixel shifting mechanism that shifts the relative position between the subject image and the image sensor 120 on a plane perpendicular to the optical axis. The pixel shifting mechanism will be described later.

[0029] The array of image signals acquired by the image sensor 120 is converted into digital signals by an A / D converter (not shown) and acquired as image data. The image data is transferred to the DRAM 160 via the bus 150 and temporarily stored therein.

[0030] In the imaging device 100 of this embodiment, the imaging element 120 is moved in a direction perpendicular to the optical axis by the shift mechanism 130, which is a pixel shifting mechanism, and acquires image data at each of the positions to which the imaging element 120 is moved. The multiple pieces of image data acquired in this way are temporarily stored in the DRAM 160.

[0031] The image data of the multiple frames temporarily stored in the DRAM 160 are transferred to the pixel shifting synthesis processing unit 140 via the bus 150. The pixel shifting synthesis processing unit 140 generates one image data with a high resolution by synthesis processing using the image data of the multiple frames. The pixel shifting synthesis processing unit 140 will be described later.

[0032] Finally, the image data of one frame having been made high resolution is transferred to the external memory 170 via the bus 150 and stored therein.

[0033] (Figure 1: Stacked image sensor) Next, an imaging element having a stacked structure will be described. An imaging element having a stacked structure is an imaging element configured by stacking photodiodes, which are photoelectric conversion units. In particular, in this embodiment, an imaging element having a stacked structure is adopted, in which photodiodes, which are photoelectric conversion units, are stacked in three layers in the depth direction.

[0034] (Figure 2: Three-layer image sensor) Figure 2 is a schematic diagram showing the cross-sectional structure of a three-layered imaging element. As shown in Figure 2A, each pixel constituting the three-layered imaging element is configured by stacking blue (B), green (G), and red (R) photodiodes in order in the depth direction. This makes it possible to directly obtain pixel values ​​corresponding to each color in each pixel.

[0035] In addition, as shown in Fig. 2B, the array of pixel values ​​corresponding to each color acquired at each pixel is acquired as image data of each layer corresponding to each color. In this embodiment, a set of image data of each layer acquired using an imaging element with a three-layer structure is called one piece of image data.

[0036] Semiconductors such as silicon used as materials for photodiodes have the property that the absorption rate varies depending on the penetration depth for each wavelength component of the incident light. Due to this property of semiconductors, among the wavelength components of the incident light, the shorter the wavelength component, the shallower the absorption depth of the semiconductor, and the longer the wavelength component, the deeper the absorption depth of the semiconductor.

[0037] A stacked image sensor utilizes this semiconductor property, and by stacking photodiodes made of silicon or the like, it is possible to vertically separate the incident light into each wavelength component. This allows the stacked image sensor to directly obtain pixel values ​​corresponding to each color at each pixel. Therefore, compared to an image sensor with color filters, a stacked image sensor does not require a color filter for color separation of the incident light above the photodiode, and does not require demosaic processing, which is a cause of false colors when acquiring image data.

[0038] The imaging device 100 of this embodiment uses an imaging element with a stacked structure for the imaging element 120, so that the number of pieces of image data to be acquired by pixel shifting can be reduced.

[0039] In the color-filtered image sensor, a color filter for acquiring a pixel value corresponding to each color is arranged for each pixel. The color filters arranged for each pixel are regularly arranged two-dimensionally on the plane of the color-filtered image sensor, for example, in a Bayer array.

[0040] When acquiring image data using an image sensor with a color filter, first, pixel values ​​corresponding to each color are acquired for each pixel, and the pixel values ​​of each pixel are acquired as an array. Then, image data is generated by performing demosaic processing on the array of pixel values ​​of each pixel.

[0041] Similarly, when acquiring multiple pieces of image data by pixel shifting using an image sensor with color filters, it is necessary to construct an array of pixel values ​​on the premise of demosaic processing. However, in order to construct an array of pixel values ​​on the premise of demosaic processing from multiple pieces of image data, it is necessary to acquire a large number of pieces of image data.

[0042] In contrast, a stacked image sensor can directly obtain pixel values ​​corresponding to each color for each pixel. When acquiring multiple images of data by pixel shifting using a stacked image sensor, unlike when an image sensor with color filters is used, there is no need to configure an array of pixel values ​​that assumes demosaic processing.

[0043] Therefore, when acquiring multiple pieces of image data by pixel shifting using an imaging element with a stacked structure, the number of pieces of image data to be acquired can be reduced by pixel shifting; by acquiring at least two pieces of image data and performing a synthesis process on them, it is possible to generate image data with a higher resolution than normal image data.

[0044] In particular, the imaging device 100 of this embodiment acquires two pieces of image data by pixel shifting using an imaging element with a three-layer structure, and generates image data with a higher resolution than normal image data by performing a synthesis process on these pieces of image data.

[0045] (Figure 1: Pixel shift mechanism) Next, the pixel shifting mechanism will be described. In this embodiment, a shift mechanism 130 that shifts the image sensor 120 on a plane perpendicular to the optical axis is adopted as the pixel shifting mechanism. The shift mechanism 130 can shift the relative position between the subject image and the image sensor 120 on a plane perpendicular to the optical axis. The driving means of the shift mechanism 130 is, for example, a voice coil motor.

[0046] The shift mechanism 130 shifts the image sensor 120 on a plane perpendicular to the optical axis, and can perform accurate position control to a target position. The target position is determined based on the pixel pitch P of the image sensor 120. The image sensor 120 is positionally controlled to a plurality of target positions by the shift mechanism 130, and image data is acquired at each target position.

[0047] The pixel pitch P of the image sensor 120 is set by setting the intervals in the XY directions of each pixel that is two-dimensionally arranged at equal intervals in the image sensor 120 as PX and PY. In this embodiment, the pixel pitch P in the XY directions is particularly represented as (PX, PY). With regard to the pixel pitch P in the XY directions, PX and PY may be different intervals.

[0048] The pixel shifting mechanism may also be a shift mechanism that shifts all or part of the photographing lens on a plane perpendicular to the optical axis. By shifting all or part of the photographing lens on a plane perpendicular to the optical axis, the relative position between the subject image and the image sensor 120 can also be shifted on the plane perpendicular to the optical axis.

[0049] (Figure 1: Pixel shift synthesis processing unit) Next, a description will be given of the pixel shift synthesis processing unit 140. In the pixel shift synthesis processing unit 140 in Fig. 1, reference numeral 141 denotes an image data input unit, 142 denotes an image data arrangement unit, 143 denotes a resampling unit, 144 denotes a pixel mixing unit, and 145 denotes an image data output unit.

[0050] The image data input unit 141 inputs a plurality of pieces of image data acquired by pixel shifting and temporarily stored from the DRAM 160 to the pixel shifting synthesis processing unit 140. In this embodiment, two pieces of image data are input from the DRAM 160 to the pixel shifting synthesis processing unit 140.

[0051] The image data arrangement unit 142 arranges the input image data for the plurality of frames by shifting them based on the pixel pitch P. In this embodiment, an image sensor with a three-layer structure is adopted for the image sensor 120, and image data for each layer is acquired for one image data. Therefore, the image data arrangement unit 142 arranges the image data for each layer for the input image data for the plurality of frames by shifting them. The method of arranging the input image data for the plurality of frames by the image data arrangement unit 142 will be described later.

[0052] The resampling unit 143 regenerates image data of each layer, which has been made higher resolution or lower resolution, from the image data of each layer arranged with a shift, by performing a resampling process. The resampling process by the resampling unit 143 will be described later.

[0053] The pixel mixer 144 generates image data for recording by pixel mixing processing using the image data of each layer regenerated by the resampling processing. The pixel mixing processing by the pixel mixer 144 will be described later.

[0054] The image data output unit 145 outputs the image data for recording generated by the pixel mixing process. The output image data for recording is recorded in the external memory 170, transferred to an image compression unit (not shown) for image compression, or transferred to an image processing unit (not shown) for other post-processing image processing.

[0055] (Figure 3: How to arrange multiple image data) Next, a method for arranging a plurality of input image data by the image data arrangement unit 142 will be described.

[0056] The imaging device 100 of this embodiment employs a three-layered imaging element for the imaging element 120, and acquires two pieces of image data by pixel shifting. The image data arrangement unit 142 arranges the image data of each layer constituting each of the two pieces of image data by shifting them based on the pixel pitch P.

[0057] Fig. 3 is a schematic diagram showing a method of arranging two pieces of image data acquired by pixel shifting by the image data arrangement unit 142. In Fig. 3, image data A and image data B respectively show two pieces of image data acquired by pixel shifting.

[0058] A single image data piece captured by a three-layered imaging element is composed of image data for the B layer (blue), G layer (green), and R layer (red). That is, in Fig. 3, image data A is composed of image data AB for the B layer, image data AG for the G layer, and image data AR for the R layer. Image data B is composed of image data BB for the B layer, image data BG for the G layer, and image data BR for the R layer.

[0059] 3, the image data arrangement unit 142 arranges and shifts the image data of each layer constituting each of the image data A and image data B. At this time, the image data arrangement unit 142 arranges and shifts the image data of each layer constituting each of the image data A and image data B based on the pixel pitch P so that the pixel positions of the pixels do not overlap each other.

[0060] Fig. 4 is a schematic diagram showing how the image data of each layer constituting each of the image data A and image data B are arranged with a shift based on the pixel pitch P. Fig. 4 shows the image data of one layer of the image data of each layer of the image data A and image data B, and the image data of the other layers are similar to Fig. 4. The schematic diagram of Fig. 4 is represented typically by the central positions of each pixel constituting the image data of each layer, which are two-dimensionally arranged on a plane. That is, in Fig. 4, the arrangement of the central positions of each pixel indicates the arrangement of each pixel constituting the image data of each layer.

[0061] As shown in Fig. 4, the image data of each layer constituting each of the image data A and image data B is shifted based on the pixel pitch P so that the center positions of the pixels constituting the image data of each layer do not overlap with each other. In this embodiment, the image data are shifted by 1 / 2 the pixel pitch P. That is, the image data are shifted by (1 / 2PX, 1 / 2PY), which is 1 / 2 the pixel pitch P, in the XY directions in each layer. The amount of shift in each layer is not limited to this embodiment, and by setting it to a non-integer multiple of the pixel pitch P, the image data of each layer constituting each of the image data A and image data B can be shifted and arranged so that the pixel centers of the pixels do not overlap with each other.

[0062] If the image data of each layer constituting each of image data A and image data B are shifted so that the center positions of each pixel overlap each other, the pixel values ​​of each pixel constituting the image data of each layer will simply be added together. In this case, it will be impossible to regenerate the high-resolution image data by resampling processing.

[0063] (Figure 1: Resampling process) Next, the resampling process will be described. In this embodiment, the resampling process refers to regenerating the image data of each layer by defining new pixel positions in the image data of each layer that is shifted and calculating pixel values ​​at the newly defined pixel positions by interpolation.

[0064] In this embodiment, a three-layered imaging element is used for the imaging element 120, and two pieces of image data are acquired by pixel shifting. Then, the image data of each layer constituting each of the acquired two pieces of image data is regenerated by resampling processing from the image data of each layer arranged with a shift. At this time, the image data of each layer arranged with a shift has twice the amount of information of the image data of each layer constituting one piece of image data, so theoretically it is possible to regenerate image data of each layer with a resolution up to twice as high as the image data of each layer arranged with a shift.

[0065] Here, in this embodiment, a stacked structure imaging element (a three-layer structure imaging element) is used for the imaging element 120, and problems that arise when multiple (two) pieces of image data are acquired by pixel shifting will be described.

[0066] In the structure of the stacked imaging element, the photoelectric conversion units that convert the subject image into an image signal are arranged in a stacked manner. Therefore, one piece of image data acquired by the stacked imaging element is composed of image data of each layer corresponding to the stacked photoelectric conversion units. Furthermore, the resolution of the image data of each layer is equivalent to the resolution of one piece of image data acquired by the imaging element with a color filter. Therefore, the amount of information of the image data acquired by the stacked imaging element is much greater than the amount of information of the image data acquired by the imaging element with a color filter.

[0067] Furthermore, when a stacked imaging element is used and multiple pieces of image data are acquired by shifting pixels, the amount of information in the image data to be handled becomes even larger.

[0068] When the amount of image data handled increases, the burden of transferring the image data and the burden of processing the image data in subsequent processes also increase, and there is also the problem that the capacity of the buffer memory for temporarily storing the image data must be increased.

[0069] In this embodiment, an imaging element with a stacked structure is adopted, and the image data of each layer that constitutes multiple image data acquired by pixel shifting is divided into a luminance layer and a color layer. Image data of different resolutions for each layer are regenerated by a resampling process, thereby preventing the amount of information in the image data from increasing.

[0070] Since human visual characteristics are more sensitive to luminance information than to color information, the luminance signal, among the image signals constituting image data, has a greater impact on human resolution than the color signal. In this embodiment, in the resampling process, the image data of each layer is divided into a luminance layer and a color layer, and the amount of information of the image data regenerated in the luminance layer is increased while the amount of information of the image data regenerated in the color layer is reduced, thereby suppressing the amount of information of the image data while maintaining the resolution of the image data.

[0071] Next, a method of dividing the image data of each layer, which is shifted and arranged in the resampling process, into a luminance layer and a color layer will be described. In this embodiment, a three-layered imaging element is adopted, and the image data of each layer constituting two pieces of image data acquired by pixel shifting is shifted and arranged, and one layer of the image data of each layer is a luminance layer and the other layers are color layers. In particular, in this embodiment, the B layer of the image data of each layer is a luminance layer, and the G layer and the R layer are color layers. The method of dividing the image data of each layer into a luminance layer and a color layer is not limited to the method of this embodiment.

[0072] Also, instead of dividing the image data of each layer into a luminance layer and a color layer in advance as in the method of this embodiment, the image data of each layer may be divided by evaluating the image data of each layer obtained. For example, among the image data of each layer, one layer from which image data with the highest contrast is obtained may be the luminance layer, and the other layers may be the color layers. Also, among the image data of each layer, one layer from which image data with the highest signal value is obtained may be the luminance layer, and the other layers may be the color layers.

[0073] In this embodiment, in order to reduce the amount of information in the image data, the image data of each layer is divided into a luminance layer and a color layer, and in the resampling process, image data with a higher resolution than the normal image data of each layer is regenerated in the luminance layer from which luminance information is obtained, and image data with a lower resolution than the normal image data of each layer is regenerated in the color layer from which color information is obtained.

[0074] The resolution of the image data of each layer regenerated by the resampling process in the luminance layer and the chrominance layer is determined by the pixel pitch P' of the image data of each layer regenerated by the resampling process. The pixel pitch P' is determined based on the normal pixel pitch P of the image data of each layer (the pixel pitch P of the image sensor 120).

[0075] If the pixel pitch P' of the regenerated image data is smaller than the pixel pitch P of the normal image data of each layer, high-resolution image data is regenerated, and if it is larger, low-resolution image data is regenerated.

[0076] In this embodiment, the pixel pitch P' of the image data regenerated by the resampling process is set to 1 / √2 times the pixel pitch P in the luminance layer, and 2 / √2 times the pixel pitch P in the chrominance layers. As a result, image data with a resolution twice as high as the normal image data of each layer is regenerated in the luminance layer, and image data with a resolution half as low as the normal image data of each layer is regenerated in the chrominance layers.

[0077] In this embodiment, by making the resolution of the image data regenerated by the resampling process different for the luminance layer and the chrominance layer for the image data of each shifted layer, it is possible to reduce the amount of information in the entire image data when obtaining image data with increased resolution by pixel shifting while adopting an imaging element with a stacked structure.

[0078] In particular, in this embodiment, the resolution of the image data reproduced in the luminance layer is increased to increase the amount of information, while the resolution of the image data reproduced in the chrominance layer is decreased to reduce the amount of information, thereby reducing the amount of information in the overall image data, including the image data of the luminance layer and the chrominance layer.

[0079] Next, a method for regenerating image data of each layer by resampling processing in the luminance layer and the chrominance layer will be described.

[0080] First, in the image data of each layer that is shifted, the pixel position of each pixel of the image data regenerated by the resampling process is newly defined based on the pixel pitch P of the normal image data of each layer.

[0081] Next, neighboring pixels to be used in calculating pixel values ​​at pixel positions of each newly defined pixel are determined. The neighboring pixels are determined to include pixels that are at a predetermined distance from the pixel positions of each newly defined pixel in the image data of each layer that are shifted.

[0082] Next, at the pixel position of each newly defined pixel, the pixel value is calculated using the determined surrounding pixels. The pixel value is calculated by an interpolation calculation using the pixel values ​​of the determined surrounding pixels. As the method of the interpolation calculation, a conventional method such as a bicubic method or a bilinear method may be appropriately adopted.

[0083] Then, the image signal array of the pixel values ​​calculated at all of the pixel positions of each newly defined pixel is regenerated as image data for each layer.

[0084] Next, the method of resampling the luminance and chrominance layers will be described.

[0085] (Figure 5: How the luminance layer is resampled) The method of the luminance layer resampling process will be described with reference to Fig. 5. Fig. 5 is a schematic diagram showing how an imaging element with a three-layer structure is adopted, the image data of the luminance layers constituting each of two pieces of image data (image data A and image data B) acquired by pixel shifting are shifted and arranged, and image data with a higher resolution than the normal image data of each layer is regenerated from the shifted image data of the luminance layers by resampling process.

[0086] First, as shown in FIG. 5A, in the image data of the luminance layer arranged in a shifted manner, the pixel position of each pixel of the image data regenerated by the resampling process is newly defined.

[0087] The pixel positions newly defined in the luminance layer are determined based on the pixel pitch P of the image data of each normal layer. In order to regenerate image data with a higher resolution than the image data of each normal layer in the luminance layer, the pixel pitch P'L of the image data regenerated by resampling processing in the luminance layer is set so that P'L < P. In this embodiment, in order to regenerate image data with a resolution twice as high in the luminance layer, it is set so that P'L = 1 / √2P.

[0088] As shown in FIG. 5A, within the range of the image data of the luminance layer arranged with a shift, the pixel positions of each pixel are newly defined based on the pixel pitch P'L. The center position of the leftmost and uppermost pixel of the image data arranged with a shift in the luminance layer is set as the origin (0, 0), and starting from this origin (0, 0), the pixel positions are newly defined in the XY directions based on the pixel pitch P'L (1 / √2PX, 1 / √2PY). Thereby, the pixel positions of each pixel constituting the image data of the luminance layer regenerated by resampling processing are newly defined.

[0089] In the luminance layer, the number of pixels in the XY directions of the image data composed of each pixel with a newly defined pixel position is √2 times that of the image data of each normal layer respectively. Also, the resolution is increased to twice as high while keeping the aspect ratio the same as that of the image data of each normal layer. Note that since the number of pixels in the XY directions constituting the image data must actually be an integer, the number of pixels in the XY directions needs to be appropriately replaced with the number of pixels obtained by rounding off the fractional part so that they are integers respectively.

[0090] Next, as shown in FIG. 5B, in the luminance layer, the peripheral pixels used for calculating the pixel values at the pixel positions of each newly defined pixel are determined.

[0091] For each pixel position of each newly defined pixel, the peripheral pixels are determined so as to include the pixels at a predetermined distance among the pixels constituting the image data of the luminance layer arranged with a shift.

[0092] In this embodiment, the peripheral pixels for the pixel positions of each newly defined pixel in the luminance layer are determined by the following formula. By arbitrarily setting k in the following formula to define the range of the peripheral pixels in the luminance layer so as to include one or more peripheral pixels, the predetermined distance from the newly defined pixel position can be adjusted. Note that the method for determining the peripheral pixels is not limited to the following formula. |x - x'| + |y - y'| < k (k is an arbitrary positive number) (x, y): Coordinates of the newly defined pixel position (x', y'): Coordinates of the pixel position of the peripheral pixel

[0093] Next, for the pixel positions of each newly defined pixel in the luminance layer, the pixel value is calculated using the peripheral pixels.

[0094] The pixel value at the pixel position of each newly defined pixel is calculated by interpolation using the pixel values of the peripheral pixels. As the method of interpolation, conventional interpolation methods such as the bicubic method and the bilinear method may be appropriately adopted. By calculating the pixel values at all the pixel positions for each newly defined pixel, the image data with higher resolution in the luminance layer can be regenerated.

[0095] (Fig. 6: Method of resampling process for color layer) The method of the resampling process for the color layer will be described with reference to Fig. 6. Fig. 6 is a schematic diagram showing that the image data of the color layers constituting two pieces of image data (image data A and image data B) obtained by pixel shift are arranged with a shift in a three-layer structure image sensor, and image data with lower resolution than the image data of each normal layer is regenerated by resampling from the arranged image data of the color layers with a shift.

[0096] First, as shown in Fig. 6A, in the image data of the color layers arranged with a shift, the pixel positions of each pixel of the image data to be regenerated by the resampling process are newly defined.

[0097] The pixel positions newly defined in the color layer are determined based on the pixel pitch P of the normal image data of each layer. In order to regenerate image data in the color layer with a lower resolution than the normal image data of each layer, the pixel pitch P'C of the image data regenerated by resampling processing in the color layer is set so that P'C > P. In this embodiment, in order to regenerate image data in the color layer with a resolution reduced by half, P'C is set so that P'C = 2 / √2P.

[0098] As shown in Fig. 6A, the pixel position of each pixel is newly defined based on the pixel pitch P'C within the range of the image data of the color layers that are shifted. The center position of the top left pixel of the image data that are shifted in the color layers is set as the origin (0,0), and a pixel position (1 / 2√2PX, 1 / 2√2PY) that is moved 1 / 2√2 of the pixel pitch P(PX,PY) in the XY directions from this origin (0,0) is set as the starting point, and a new pixel position is defined in the XY directions based on the pixel pitch P'C (2 / √2PX, 2 / √2PY). This newly defines the pixel position of each pixel that constitutes the image data of the color layers that are regenerated by the resampling process.

[0099] 6B, the pixel positions of the pixels newly defined in the chrominance layer are defined one for every four pixel positions of the pixels newly defined in the luminance layer. In other words, the resolution of the image data regenerated in the chrominance layer is 1 / 4 times that of the image data regenerated in the luminance layer.

[0100] In the color layer, the number of pixels in the X and Y directions of the image data consisting of each pixel with a newly defined pixel position will be 1 / √2 times that of the normal image data of each layer. Also, the resolution is reduced to 1 / 2 while keeping the aspect ratio the same as that of the normal image data of each layer. Note that since the number of pixels in the X and Y directions that make up the image data must actually be integers, the number of pixels in the X and Y directions must be appropriately replaced with the number of pixels that has been rounded down to an integer.

[0101] Next, as shown in FIG. 6C, in the color layer, neighboring pixels to be used in calculating pixel values ​​at the pixel positions of each newly defined pixel are determined.

[0102] The surrounding pixels are determined so as to include pixels at a predetermined distance from the pixel position of each newly defined pixel, among the pixels constituting the image data of the color layers arranged in a shifted manner.

[0103] In this embodiment, the pixel pitch P'C of the pixel positions of each pixel newly defined in the color layer is larger than the pixel pitch P'L of the pixel positions of each pixel newly defined in the luminance layer, so the surrounding pixels of the color layer also include pixels that are located at a greater distance from the pixel positions of each newly defined pixel.

[0104] In this embodiment, the neighboring pixels for each pixel position newly defined in the color layer are determined by the following formula. The predetermined distance from the newly defined pixel position can be adjusted by arbitrarily setting 2k in the following formula to define the range of neighboring pixels in the color layer so as to include one or more neighboring pixels. Note that the method of determining the neighboring pixels is not limited to the following formula. |x-x'|+|y-y'|<2k (k is any positive number) (x,y): The coordinates of the newly defined pixel position (x', y'): Coordinates of the pixel positions of the surrounding pixels

[0105] Next, for each newly defined pixel location in the color layer, a pixel value is calculated using the surrounding pixels.

[0106] The pixel value at the pixel position of each newly defined pixel is calculated by an interpolation operation using the pixel values ​​of the surrounding pixels. As the method of the interpolation operation, a conventional interpolation operation method such as a bicubic method or a bilinear method may be appropriately adopted. By calculating the pixel values ​​at all pixel positions of each newly defined pixel, the image data with reduced resolution in the color layer can be regenerated.

[0107] (Figure 1: Pixel blending process) Next, the pixel mixing process performed by the pixel mixer 144 will be described.

[0108] In the pixel mixing process, the image data of the luminance layer regenerated by the resampling process and the image data of the chrominance layers are mixed together to generate image data having a higher resolution than the normal image data of each layer.

[0109] In this embodiment, pixel mixing processing refers to image signal processing in which image data for each layer, each of which has a different resolution and is regenerated by a resampling process, is subjected to noise reduction processing and color separation processing for the image data for each layer, and the image data is finally synthesized after aligning the resolutions to generate high-resolution image data.

[0110] A control method for the image pickup apparatus 100 of this embodiment will be described with reference to Fig. 7 to Fig. 10. The photographing operation in the control method for the image pickup apparatus 100 of this embodiment will be described below. Furthermore, among various photographing operations, the description will be limited to a photographing operation that generates image data with a high resolution finally by pixel shifting.

[0111] (Figure 7: Each step of the shooting process) Each step of the shooting operation will be described with reference to FIG.

[0112] First, in S001, the control unit of the imaging device 100 detects whether the release switch is turned ON. If the release switch is not turned ON, the process returns to S001, and again detects whether the release switch is turned ON. If the release switch is turned ON, the focusing operation is then performed.

[0113] Next, in S002, the control unit of the imaging device 100 performs a focusing operation. The control unit of the imaging device 100 drives the focus lens group included in the photographing lens 110 to focus on a subject based on a focusing evaluation value detected by a contrast detection method, a phase difference detection method, or the like.

[0114] Next, in S003, the control unit of the imaging device 100 acquires two pieces of image data by pixel shifting. The pixel shifting is performed by the control unit of the imaging device 100 shifting the imaging element 120 (an imaging element having a three-layer structure) in a plane perpendicular to the optical axis by the shift mechanism 130. The imaging element 120 acquires one piece of image data at each of two positions, a first position and a second position, shifted by pixel shifting. In this embodiment, the first position is the origin position (0,0), and the second position is a position (1 / 2PX, 1 / 2PY) shifted from the origin position (0,0) by 1 / 2 the pixel pitch P (PX, PY) in the XY direction. Each step of acquiring image data by pixel shifting in S003 will be described later.

[0115] Next, in S004, the pixel shift synthesis processing unit 140 performs pixel shift synthesis processing using two pieces of image data acquired at the first position and the second position by pixel shifting. Here, the acquired two pieces of image data are acquired by an imaging element with a three-layer structure, and are each composed of image data of each layer (B layer, G layer, and R layer). The pixel shift synthesis processing is performed by regenerating image data with a higher or lower resolution than normal image data of each layer from the image data of each layer that is shifted and arranged by the resampling processing, and finally generating image data with a higher resolution using the image data of each layer regenerated by the pixel mixing processing. Each step of the pixel shift synthesis processing in S004 will be described later.

[0116] Next, in S005, the finally generated high-resolution image data is recorded. The finally generated high-resolution image data is recorded in the external memory 170 for storage, or is recorded in a temporary memory inside the image sensor for other image processing.

[0117] (Figure 8: Acquiring image data by pixel shifting S003) Next, each step of acquiring image data by pixel shifting (FIG. 7: S003) will be described with reference to FIG.

[0118] First, in S301, the control unit of the imaging device 100 moves the imaging element 120 to the origin position (0,0), which is a first position, by pixel shifting. Next, in S302, the imaging element 120 is exposed at the first position to obtain first image data, and in S303, the obtained first image data is temporarily stored in the DRAM 160.

[0119] Next, in S304, the control unit of the imaging device 100 moves the imaging element 120 by pixel shifting to a position (1 / 2PX, 1 / 2PY) shifted in the XY direction from the origin position (0,0) which is the second position by 1 / 2 the pixel pitch P (PX, PY). Next, in S305, the imaging element 120 is exposed at the second position to obtain second image data, and in S306, the obtained second image data is temporarily stored in the DRAM 160. At this time, the DRAM 160 has recorded therein the image data for two frames obtained by pixel shifting.

[0120] In this embodiment, when acquiring image data by pixel shifting, in order to prevent the pixels that make up the image data acquired at each position from overlapping with each other, the image sensor 120 is moved to a position that is shifted relatively by 1 / 2 the pixel pitch P(PX, PY) at each position.

[0121] (Figure 9: Pixel shift synthesis process S004) Next, each step of the pixel shift synthesis process (FIG. 7: S004) will be described with reference to FIG.

[0122] First, in S 401 , the pixel shift synthesis processing unit 140 inputs two pieces of image data recorded in the DRAM 160 by the image data input unit 141 .

[0123] Next, in S402, the pixel shift synthesis processing unit 140 separates the image data of each layer constituting the two image data into a luminance layer and a color layer. In this embodiment, the image data of the B layer is set as the luminance layer, and the image data of the G layer and the R layer are set as the color layers in advance.

[0124] Next, in S403, the pixel shift synthesis processing unit 140 shifts and arranges the image data of the luminance layer and the chrominance layer constituting each of the two pieces of image data by the image data arrangement unit 142. At this time, the image data arrangement unit 142 arranges the image data of each layer by relatively shifting it by 1 / 2 the pixel pitch P(PX, PY) in the same way as the pixel shift.

[0125] Next, in S404, pixel shift synthesis processing unit 140 regenerates image data of each layer having a different resolution from the normal image data of each layer from the image data of each layer arranged with a shift in the luminance layer and the chrominance layer by resampling unit 143. At this time, image data with a higher resolution than the normal image data of each layer is regenerated in the luminance layer, and image data with a lower resolution than the normal image data is regenerated in the chrominance layer.

[0126] Next, in S405, the pixel shift synthesis processing unit 140 uses the regenerated luminance layer or chrominance layer image data to generate final high-resolution image data by the pixel mixing unit 144. Each step of the pixel mixing process in S405 will be described later.

[0127] (Figure 10: Pixel mixing process S405) Next, each step of the pixel mixture process (FIG. 9: S405) will be described with reference to FIG.

[0128] In the following description, a single image data consisting of image data of a luminance layer and a chrominance layer will be expressed using the ratio of the resolution of the image data of each layer. The ratio of the resolution of the image data of each layer will be expressed in the order of R layer, G layer, and B layer.

[0129] In this embodiment, the pixel shift synthesis processing unit 140 divides the image data of each layer constituting one image data into the B layer as the luminance layer, and the G layer and the R layer as the color layers in advance. The resolution ratio of the image data of the luminance layer and the color layer regenerated by the resampling process is 4:1.

[0130] Therefore, for example, a piece of image data composed of image data of the luminance layer and the chrominance layer regenerated by the resampling process can be expressed as 114 image data using the ratio of the resolution of the image data of each layer. Also, for example, a piece of image data obtained by deleting the image data of the chrominance layers R and G from the 114 image data can be expressed as 004 image data.

[0131] First, in S501, the pixel shift synthesis processing unit 140 generates 111 image data and 004 image data from the 114 image data regenerated by the pixel mixing unit 144 through resampling processing.

[0132] Here, the 111 image data is image data composed of image data of the R and G layers of the 114 image data, and image data of the B layer obtained by adding together pixel values ​​of a set of four adjacent pixels of the 114 image data. Also, the 004 image data is image data composed only of the image data of the B layer of the 114 image data, with the image data of the R and G layers deleted from the 114 image data.

[0133] Next, in S502, 111 image data and 001 image data are further generated from the 111 image data generated in S501.

[0134] Here, the 001 image data is image data obtained by deleting the image data of the R and G layers from the 111 image data, and is composed of only the image data of the B layer of the 111 image data.

[0135] Next, in S503, noise reduction processing and color separation processing are performed on the 111 image data generated in S502, and further, in S504, upsampling processing is performed on the processed 111 image data to generate 444 image data.

[0136] The image data of each layer constituting the 444 image data generated in S504 corresponds to image data whose resolution is twice as high as that of normal image data of each layer.

[0137] On the other hand, in S505, noise reduction processing is performed on the 004 image data generated in S501. Also, in S506, upsampling processing is performed on the 001 image data generated in S502 to generate 004 image data.

[0138] Then, in S507, the difference between the 004 image data processed in S505 and the 004 image data generated in S506 is calculated to generate differential image data.

[0139] Finally, in S508, the differential image data generated in S507 is reflected on the 444 image data generated in S504, and 444 image data is generated that is composed of image data of each layer that is twice as high resolution as the normal image data of each layer.

[0140] In the pixel mixing process of this embodiment, 111 image data and 004 image data are generated from 114 image data composed of image data of luminance layer and chrominance layer regenerated by resampling process. Then, 444 image data is generated from 111 image data by performing noise reduction process and upsampling process, and 004 image data is generated from 004 image data while performing noise reduction process without upsampling process. Furthermore, by reflecting 004 image data as difference image data to 444 image data, 444 image data composed of image data of each layer with twice the resolution of normal image data of each layer is finally generated. This makes it possible to prevent the problem of coloring of image data that occurs when performing upsampling process after performing noise reduction process.

[0141] As described above, according to the present invention, in an imaging device and a control method for an imaging device that acquires image data with a higher resolution by pixel shifting, the number of multiple pieces of image data to be acquired by pixel shifting can be reduced by adopting an imaging element with a stacked structure. This also reduces the burden of data transfer and image processing of multiple pieces of image data, and reduces the capacity of a buffer memory that temporarily stores multiple pieces of image data.

[0142] In addition, by adopting a stacked imaging element and performing a resampling process on the image data of each layer that constitutes multiple image data acquired by pixel shifting, the image data of each layer is divided into a luminance layer and a color layer, and the resolution of the image data regenerated in the luminance layer and the color layer is made different, thereby reducing the amount of information in the overall image data. [Explanation of symbols]

[0143] 100 Imaging device 110 Photographic Lens 120 Image sensor 130 Shift mechanism 140 Pixel shift synthesis processing unit 141 Image data input unit 142 Image data placement section 143 Resampling section 144 pixel mixing section 145 Image data output unit 150 Bus 160 DRAM 170 External Memory

Claims

1. An imaging device that generates image data having a higher resolution than normal image data by a synthesis process using a plurality of image data acquired by pixel shifting, The imaging device includes a photographing lens that forms a subject image on an imaging plane from subject light, an imaging element having a laminated structure in which photoelectric conversion units that convert the subject image into an image signal are laminated, a pixel shifting mechanism that shifts the relative position of the subject image and the imaging element having the laminated structure on a plane perpendicular to an optical axis, and a pixel shift synthesis processing unit that generates one image data by synthesis processing from a plurality of image data acquired by pixel shifting, an image pickup device characterized in that the pixel shifting synthesis processing unit includes an image data input unit that inputs two pieces of image data acquired by pixel shifting; an image data arrangement unit that arranges the image data of each layer that constitutes each of the two pieces of image data with a shift; a resampling unit that regenerates image data of each layer from the image data of each layer arranged with a shift by a resampling process using image data of the same layer; a pixel mixing unit that generates one piece of image data by pixel mixing process using the regenerated image data of each layer; and an image data output unit that outputs the one piece of image data generated by the pixel mixing process.

2. 2. The imaging device according to claim 1, wherein the pixel shifting mechanism shifts a relative position between an image of a subject and the imaging element of the stacked structure in a plane perpendicular to the optical axis by moving the imaging element of the stacked structure in a direction perpendicular to the optical axis.

3. 2. The imaging device according to claim 1, wherein the pixel shifting mechanism shifts a relative position between an image of a subject and the image sensor of the stacked structure on a plane perpendicular to the optical axis by moving all or a part of the photographing lens in a direction perpendicular to the optical axis.

4. 4. The imaging device according to claim 1, wherein the pixel shifting mechanism shifts the relative position of the subject image and the stacked imaging element by 1 / 2 the pixel pitch of the stacked imaging element in the XY directions on a plane perpendicular to the optical axis.

5. the pixel shift synthesis processing unit divides image data of each layer constituting each of the two pieces of image data acquired by pixel shifting using the stacked image sensor into a luminance layer and a color layer; An imaging device as described in any one of claims 1 to 4, characterized in that by a resampling process, image data having a higher resolution than normal image data is regenerated in the luminance layer, and image data having a lower resolution than normal image data is regenerated in the chrominance layer.

6. An imaging device that generates image data having a higher resolution than normal image data by a synthesis process using multiple image data acquired by pixel shifting, The imaging device includes a photographing lens that forms a subject image on an imaging plane from subject light, an imaging element having a laminated structure in which photoelectric conversion units that convert the subject image into an image signal are laminated, a pixel shifting mechanism that shifts the relative position of the subject image and the imaging element having the laminated structure on a plane perpendicular to an optical axis, and a pixel shift synthesis processing unit that generates one image data by synthesis processing from a plurality of image data acquired by pixel shifting, the pixel shift synthesis processing unit includes an image data input unit that inputs two pieces of image data acquired by pixel shifting, an image data arrangement unit that arranges image data of each layer constituting each of the two pieces of image data with a shift, a resampling unit that regenerates image data of each layer from the image data of each layer arranged with a shift by a resampling process, a pixel mixing unit that generates one piece of image data by a pixel mixing process using the regenerated image data of each layer, and an image data output unit that outputs the one piece of image data generated by the pixel mixing process, Dividing the image data of each layer constituting each of the two pieces of image data acquired by the pixel shifting into a luminance layer and a chrominance layer; By the resampling process, image data having a higher resolution than normal image data is regenerated in the luminance layer, and image data having a lower resolution than normal image data is regenerated in the chrominance layer; An imaging device characterized in that, among the image data of each layer that constitutes each of the two image data acquired by the pixel shifting, the image data of the layer with the highest contrast is designated as a luminance layer, and the image data of the other layers is designated as color layers.

7. An imaging device that generates image data having a higher resolution than normal image data by a synthesis process using multiple image data acquired by pixel shifting, The imaging device includes a photographing lens that forms a subject image on an imaging plane from subject light, an imaging element having a laminated structure in which photoelectric conversion units that convert the subject image into an image signal are laminated, a pixel shifting mechanism that shifts the relative position of the subject image and the imaging element having the laminated structure on a plane perpendicular to an optical axis, and a pixel shift synthesis processing unit that generates one image data by synthesis processing from a plurality of image data acquired by pixel shifting, the pixel shift synthesis processing unit includes an image data input unit that inputs two pieces of image data acquired by pixel shifting, an image data arrangement unit that arranges image data of each layer constituting each of the two pieces of image data with a shift, a resampling unit that regenerates image data of each layer from the image data of each layer arranged with a shift by a resampling process, a pixel mixing unit that generates one piece of image data by a pixel mixing process using the regenerated image data of each layer, and an image data output unit that outputs the one piece of image data generated by the pixel mixing process, Dividing the image data of each layer constituting each of the two pieces of image data acquired by the pixel shifting into a luminance layer and a chrominance layer; By the resampling process, image data having a higher resolution than normal image data is regenerated in the luminance layer, and image data having a lower resolution than normal image data is regenerated in the chrominance layer; An imaging device characterized in that, among the image data of each layer that constitutes each of the two image data acquired by the pixel shifting, the image data of the layer with the highest signal value is designated as a luminance layer, and the image data of the other layers is designated as color layers.

8. A method for controlling an imaging device that generates image data having a higher resolution than normal image data by a synthesis process using multiple pieces of image data acquired by pixel shifting, comprising the steps of: A method for controlling an imaging device, comprising: an image data acquisition step for acquiring two pieces of image data by pixel shifting using an imaging element with a stacked structure; an image data input step for inputting the two pieces of acquired image data; an image data arrangement step for arranging the image data of each layer that constitutes each of the two input pieces of image data with a shift; a resampling processing step for regenerating image data of each layer using image data of the same layer from the image data of each layer that has been shifted and arranged by the resampling processing; a pixel mixing processing step for generating one piece of image data using the image data of each layer regenerated by the pixel mixing processing; and an image data output step for outputting the generated piece of image data.

Images