Imaging device and electronic apparatus

Abstract

An imaging device according to the present disclosure comprises: a movement mechanism in which pixels that generate a signal in accordance with incident light are arranged so as to conform to a two-dimensional lattice, an imaging element including a pixel array in which optical filters having mutually different characteristics are provided in a predetermined array with respect to each of the pixels is moved by different amounts in each of a first direction along the two-dimensional lattice and a second direction different from the first direction on a plane intersecting perpendicularly with the incident direction of the light, and the imaging element is moved obliquely with respect to the two-dimensional lattice; and a control unit for performing drive control of the movement mechanism and imaging control of the imaging element. The control unit causes the imaging element to perform imaging each time the imaging element is moved by the movement mechanism.

Description

Imaging devices and electronic devices

[0001] The present disclosure relates to an imaging device and an electronic device.

[0002] A technology called video see-through (pass-through) is known, which displays images captured by a camera in real time. In recent years, this pass-through technology has been increasingly adopted, especially in the field of HMDs (Head Mounted Displays).

[0003] International Publication No. 2019 / 155809 Japanese Patent Application Laid-Open No. 2021-40261

[0004] For example, in an HMD equipped with video see-through technology, in order to obtain a highly realistic display image, it is required to capture the image at high resolution and display the image captured at high resolution without delay.

[0005] Therefore, an object of the present disclosure is to provide an imaging device and electronic device that are capable of capturing high-resolution images and transferring video data captured at high resolution at a higher speed.

[0006] The imaging device according to the present disclosure comprises a moving mechanism that moves an imaging element including a pixel array in which pixels that generate signals in response to incident light are arranged according to a two-dimensional lattice and optical filters with different characteristics are provided in a predetermined arrangement for each of the pixels, on a plane that intersects perpendicularly with the incident direction of the light, in a first direction along the two-dimensional lattice and a second direction different from the first direction, and diagonally relative to the two-dimensional lattice; and a control unit that controls the drive of the moving mechanism and controls the imaging of the imaging element, wherein the control unit performs imaging using the imaging element each time the imaging element is moved by the moving mechanism.

[0007] 1 is a schematic diagram for explaining super-resolution according to an existing technology. FIG. 1 is a schematic diagram for explaining super-resolution according to an existing technology. FIG. 2 is a schematic diagram showing an example of the appearance of a sensor module applied to an imaging device according to an embodiment of the present disclosure. FIG. 3 is a perspective view showing an example of a structure in which the sensor module according to the embodiment is disassembled. FIG. 4 is a schematic diagram showing a state in which a spherical member is sandwiched between a moving body and a base unit according to an embodiment. FIG. 5 is a schematic diagram showing a more specific example of a configuration of a moving body and a base unit according to an embodiment. FIG. 6 is a schematic diagram showing a more specific example of a configuration of a moving body and a base unit according to an embodiment. FIG. 7 is a schematic diagram for explaining movement of a moving body according to an embodiment. FIG. 8 is a block diagram of an example of a general configuration of an imaging device that performs video see-through according to an existing technology. FIG. 9 is a block diagram of an example of a configuration of an imaging device that performs video see-through using super-resolution, which is applicable to an embodiment of the present disclosure. FIG. 10 is a schematic diagram for generally explaining imaging control by an imaging device according to an embodiment. FIG. 11 is a schematic diagram for explaining a display signal according to an embodiment. FIG. 12 is a block diagram of an example of a hardware configuration of an electronic device to which an imaging device according to an embodiment is applied. FIG. 13 is a schematic diagram for explaining the effect of super-resolution applicable to an embodiment. FIG. 14 is a timing chart of an example of explaining super-resolution imaging and display processing applicable to an embodiment. FIG. 1 is a schematic diagram for explaining the configuration and processing flow of an imaging device according to an embodiment. FIG. 1 is a schematic diagram for explaining stability with respect to positional movement of an image sensor (imaging element) according to an embodiment. FIG. 2 is a schematic diagram for explaining an example of control of imaging timing for an image sensor according to an embodiment. FIG. 3 is a schematic diagram for explaining pixel shift photography according to an embodiment. FIG. 4 is a schematic diagram showing spatial resolution according to a Bayer arrangement when pixel shift is not performed. FIG. 5 is a schematic diagram showing spatial resolution according to a Bayer arrangement when pixel shift is performed. FIG. 6 is a schematic diagram for explaining pixel shift photography according to an embodiment. FIG. 7 is a schematic diagram for explaining pixel shift photography according to an embodiment. FIG. 8 is a schematic diagram showing an extended example of pixel shift photography using a first pattern example according to an embodiment. FIG. 9 is a schematic diagram for explaining an example of increasing the number of steps of change in a drive signal according to an embodiment.1A and 1B are schematic diagrams for explaining the signal level of a drive signal 70 according to the number of pixel shift steps according to an embodiment. FIG. 1C are schematic diagrams showing an example in which the signal level of a drive signal is changed in a triangular wave shape according to an embodiment. FIG. 1D are schematic diagrams showing an example in which the signal level of a drive signal is changed in a triangular wave shape according to an embodiment.

[0008] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In the following embodiments, the same components are denoted by the same reference numerals, and redundant description will be omitted.

[0009] Hereinafter, embodiments of the present disclosure will be described in the following order: 1. Existing technologies related to the technology of the present disclosure 2. Embodiments of the present disclosure 2-1. Structure of an imaging device according to an embodiment of the present disclosure 2-2. Configuration and control of an imaging device according to an embodiment of the present disclosure 2-2-1. Configuration according to an embodiment 2-2-2. Control according to an embodiment 2-2-3. Example of pixel shift photography according to an embodiment 2-2-4. Specific example of pixel shift photography according to an embodiment 3. Other embodiments

[0010] (1. Existing Technology Related to Technology of the Present Disclosure) First, prior to describing the technology according to the present disclosure, existing technology related to the technology of the present disclosure will be described.

[0011] Video see-through (pass-through) is a well-known technology that displays images captured by a camera in real time. In recent years, this video see-through technology has been increasingly adopted, particularly in the field of HMDs (Head Mounted Displays). In video see-through technology, high-resolution images must be displayed without delay in order to create highly realistic images.

[0012] When shooting with a high-resolution camera, the processor's ability to process image information can be a constraint, making it difficult to transfer image data. For example, even if there is a camera (image sensor) that shoots at 4K resolution (e.g., 3840 pixels horizontally × 2160 pixels vertically) and a display that can display at 4K resolution, if image information can only be sent at up to 2K resolution (e.g., 1920 pixels horizontally × 1080 pixels vertically) per frame rate, it is not possible to display 4K resolution video without delay.

[0013] Furthermore, as image sensors become compatible with 4K resolution and the number of pixels increases, the pixel pitch becomes smaller and sensitivity may also decrease. Although binning can be used to increase sensitivity when the number of pixels is large, the narrower the pixel pitch, the greater the loss of frames between frames, which may result in reduced efficiency.

[0014] Meanwhile, a technology called super-resolution is known. In super-resolution, image information is acquired by dividing the image sensor mounted on the camera into multiple sets of pixels that are shifted by one pixel. The acquired image information is then synthesized to generate a single high-resolution image. Image information synthesis is generally performed using software processing, which can cause a time lag compared to the frame rate and can lack real-time performance.

[0015] Furthermore, a technique is also known in which an image captured at 4K resolution is divided and the divided images are alternately displayed. However, even with this technique, the process of dividing the image captured at 4K resolution may take time, which may impair real-time performance.

[0016] To solve these problems, a method has been proposed in which the pixels of the image sensor mounted on the camera are physically shifted for each frame. In this method, the amount of information is reduced by dividing each frame into two sets of images captured by shifting the position alternately, and the image information is transferred alternately for each acquired frame. The transferred pixel information is output to a display device alternately for each frame. The displayed image can be viewed as a high-resolution image due to the persistence of vision effect.

[0017] 1A and 1B are schematic diagrams illustrating super-resolution using existing technology. In the example of Fig. 1A and 1B, pixel shift photography is performed in which the pixels of an image sensor are physically shifted for each frame, and a super-resolution display image is obtained by alternately displaying sets of frames obtained by pixel shift photography.

[0018] 1A shows the processing performed during image capture in pixel shift image capture. As shown in section (a) of FIG. 1A , a pixel array 11 included in an image sensor has pixels 10 arranged in a two-dimensional lattice pattern. For example, the image sensor captures images at position Posi#1 for odd-numbered frames, and captures images at position Posi#2, which is shifted by half a pixel in both the horizontal and vertical directions, for even-numbered frames.

[0019] As shown in section (b), for example, the image sensor transfers image information captured at position Posi#1 and image information captured at position Posi#2 separately.

[0020] 1B shows the processing during display. During display, as shown in section (a), an image 21 produced by each display pixel 20 based on image information captured at position Posi# and an image 21sft produced by each display pixel 20sft based on image information captured at position Posi#2 are alternately displayed for each frame. At this time, the display positions of the images 21 and 21sft are shifted according to positions Posi#1 and Posi#2, and the images are alternately displayed for each frame.

[0021] 1B, the image 21 generated by each display pixel 20 and the image 21sft generated by each display pixel 20sft are observed as if they were being displayed simultaneously due to the afterimage effect. Therefore, it is possible to observe a super-resolution image, i.e., an image with a resolution twice that of the image captured by the image sensor.

[0022] In an embodiment of the present disclosure, in a method for capturing super-resolution images by physically shifting the pixels of an image sensor for each frame, the direction in which the pixels are shifted is devised to achieve higher resolution. Also, as a method for shifting the pixels of the image sensor, a piezoelectric element is used to move the image sensor, and magnetic force is used to control the movement of the image sensor.

[0023] (2. Embodiments of the Present Disclosure) Next, embodiments of the present disclosure will be described.

[0024] (2-1. Structure of Imaging Device According to Embodiment of Present Disclosure) First, the structure of the imaging device according to the embodiment of the present disclosure will be described. Fig. 2 is a schematic diagram showing an example of the appearance of a sensor module applied to the imaging device according to the embodiment of the present disclosure.

[0025] 2, section (a) is a perspective view showing an example of the appearance of the sensor module 1000 including the lens unit 1110. Note that in section (a) and section (b) described later, a part of the sensor module 1000 is shown semi-transparently for the sake of explanation.

[0026] In the sensor module 1000, a moving body 1200 is provided on top of a base unit 1300, and an imaging element 1100 is fixed on the moving body 1200. The moving body 1200 is moved on a plane perpendicular to the incident direction of light incident on the imaging element 1100, using a configuration described below. A lens unit 1110 is disposed above the imaging element 1100. The imaging element 1100 is connected to one end of a flexible printed circuit board 1111, and a connector 1112 is connected to the other end. A signal output from the imaging element 1100 is guided to the connector 1112 via the flexible printed circuit board 1111. Furthermore, control signals and the like for the imaging element 1100 are input to the connector 1112 and supplied to the imaging element 1100 via the flexible printed circuit board 1111.

[0027] Although not shown here, each pixel included in the image sensor 1100 is provided with a red (R), green (G), and blue (B) color filter, and is arranged in a Bayer array with a block of four pixels (2 pixels horizontally by 2 pixels vertically) as a unit. In the Bayer array, two pixels provided with green color filters, one pixel provided with a red color filter, and one pixel provided with a blue color filter are arranged so that pixels provided with color filters of the same color are not adjacent to each other.

[0028] In the following, unless otherwise specified, a pixel provided with a green color filter will be referred to as a G pixel, a pixel provided with a red color filter as an R pixel, and a pixel provided with a blue color filter as a B pixel. Red will be referred to as R color, green as G color, and blue as B color.

[0029] In FIG. 2 , section (b) is a perspective view showing an example of the external appearance of the sensor module 1000 with the lens unit 1110 removed. The base unit 1300 is provided with magnets 1301a and 1301b. The magnets 1301a and 1301b may be permanent magnets. In the example of section (b), the magnets 1301a and 1301b are respectively provided in sections partitioned in the y direction on the drawing by members of the base unit 1300. In addition, a yoke 1302a is provided on the bottom of the magnet 1301a. Although not shown in the drawing, a yoke is also provided for the magnet 1301b on the surface facing the partition.

[0030] FIG. 3 is an exploded perspective view showing an example of the structure of the sensor module 1000 according to the embodiment.

[0031] 3, mobile body 1200 has panel 1201c made of a magnetic material provided at a position corresponding to magnet 1301a on base unit 1300. Similarly, mobile body 1200 has panels 1201a and 1201b provided on the surface of y-axis facing the origin. Panels 1201a to 1201c are, for example, steel panels. Panels 1201a to 1201c may be made of any material other than steel as long as they are magnetic.

[0032] The movable body 1200 has a V-shaped groove 1202 on one side along the y-axis of the surface (second surface) facing the base part 1300. The movable body 1200 also has a recess 1203 at approximately the center of the other side along the y-axis.

[0033] Holes 1303a, 1303b, and 1303c are provided in the surface (first surface) of base portion 1300 that faces movable body 1200. Holes 1303a and 1303b are provided at positions that correspond to V-groove 1202 when movable body 1200 is correctly positioned on base portion 1300. Hole 1303c is provided at a position that corresponds to recess 1203 when movable body 1200 is correctly positioned on base portion 1300. Spherical members 1210a, 1210b, and 1210c are placed in holes 1303a, 1303b, and 1303c, respectively.

[0034] Moving body 1200 is placed relative to base portion 1300 via spherical members 1210a to 1210c. That is, spherical members 1210a and 1210b are placed in holes 1303a and 1303b, respectively, and are abutted against V-groove 1202, and are sandwiched between moving body 1200 and base portion 1300.

[0035] 4 is a schematic diagram showing how the spherical member 1210a according to the embodiment is sandwiched between the moving body 1200 and the base portion 1300. Note that the spherical member 1210b is similar to the spherical member 1210a, and therefore a description thereof will be omitted here.

[0036] 4, the opening of hole 1303a on the surface of base unit 1300 facing movable body 1200 has the shape of a conical surface 1304. In this example, the bottom of hole 1303a is cylindrical, but this is not limited to this example, and hole 1303a may have conical surface 1304 all the way to the bottom. When spherical member 1210a is placed in hole 1303, the spherical surface abuts against conical surface 1304, and its position on base unit 1300 is fixed. Furthermore, by forming the opening of hole 1303a into conical surface 1304, spherical member 1210a can freely rotate in that position.

[0037] In hole 1303a, when spherical member 1210a is placed in hole 1303a, it is sandwiched between movable body 1200 and base portion 1300, so that spherical member 1210a also abuts against the surface of V-groove 1202. Similarly, in hole 1303b, when spherical member 1210b is placed in hole 1303b, it is sandwiched between movable body 1200 and base portion 1300, so that spherical member 1210b also abuts against the surface of V-groove 1202. Therefore, in this state, movable body 1200 can move relative to base portion 1300 only along the direction of V-groove 1202.

[0038] On the other hand, recess 1203 is formed so that the height of the abutment position of spherical member 1210c when placed in hole 1303c and abutting against the upper surface of recess 1203 is equal to the height of the abutment position of spherical members 1210a and 1210b when placed in holes 1303a and 1303b, respectively, and abutting against V-groove 1202. This ensures that the opposing surfaces (first and second surfaces) of base 1300 and movable body 1200 are parallel. Furthermore, it is preferable to determine the diameter of recess 1203 so that spherical member 1210c does not abut against the edge of recess 1203 when movable body 1200 moves.

[0039] It is preferable that the spherical members 1210a to 1210c be made of a non-magnetic material, especially a ceramic with high hardness such as alumina or zirconia. Furthermore, by filling the gap between the V-groove 1202 and the spherical members 1210a and 1210b with grease or the like, vibration during operation can be suppressed and the durability of the sliding portion (V-groove 1202) can be improved.

[0040] 5 and 6 are schematic diagrams showing in more detail an example of the configuration of the moving body 1200 and the base unit 1300 according to the embodiment. Section (a) of Fig. 5 shows the moving body 1200 as seen from the top, i.e., from the side on which the image sensor 1100 is fixed, and section (b) shows a cross section A-A' in section (a). Note that section (a) shows a portion semi-transparently for the sake of explanation.

[0041] 5, a drive signal is supplied to piezoelectric element 1400 via lead wire 1401. The positions at which spherical members 1210a, 1210b, and 1210c are arranged (the positions of holes 1303a, 1303b, and 1303c) are determined so that the center of a triangle formed by the positions of spherical members 1210a, 1210b, and 1210c includes the center of gravity and drive range of moving body 1200.

[0042] In section (b) of Figure 5, magnets 1301a are arranged on both sides of piezoelectric element 1400, and yokes 1302a and 1302b are provided, respectively. Meanwhile, as described above, panels 1201c made of a magnetic material (e.g., a steel panel) are provided at positions corresponding to the magnets 1301a on moving body 1200. Therefore, moving body 1200 is attracted toward base portion 1300 by magnetic force 1305 of each magnet 1301a. This allows moving body 1200 to maintain its positional relationship in the height direction (z-axis direction) with respect to base portion 1300.

[0043] 6, section (b) shows the moving body 1200 as seen from above, and section (a) shows the B-B' cross section of section (b). Note that in section (b), for the sake of explanation, the surface of the moving body 1200 to which the imaging element 1100 is fixed is omitted.

[0044] In section (b), the piezoelectric element 1400 expands and contracts in the y-axis direction of the figure, and the end face of the protrusion 1220 of the moving body 1200 abuts against the end face of the piezoelectric element 1400 in the expansion / contraction direction. Therefore, as shown in section (b), the moving body 1200 moves in the y-axis direction of the figure in response to the expansion and contraction of the piezoelectric element 1400. Furthermore, in the base portion 1300, magnets 1301b are disposed on both sides of the protrusion 1220, and yokes 1302b are provided for each. Meanwhile, panels 1201a and 1201b, which are magnetic bodies (e.g., steel panels), are provided at positions corresponding to the magnets 1301b of the moving body 1200. Therefore, the moving body 1200 is attracted toward the piezoelectric element 1400 by the magnetic force 1306 of each magnet 1301b. As a result, the moving body 1200 moves in the y-axis direction of the figure in response to the expansion and contraction of the piezoelectric element 1400.

[0045] 7 is a schematic diagram illustrating the movement of the moving body 1200 according to the embodiment. The magnetic forces 1305 and 1306 of the magnets 1301 a and 1301 b prevent the moving body 1200 from moving in any direction other than the direction of travel (the y-axis direction in the figure) corresponding to the expansion and contraction of the piezoelectric element 1400. That is, in the embodiment, the effect that is generally obtained by using a spring is obtained by using magnetic forces.

[0046] Therefore, it is possible to suppress movement of the moving body 1200 in directions other than the traveling direction without contact, improving the durability of the contact portion and achieving quieter operation by eliminating the generation of scraping noise that accompanies the movement of the moving body 1200. Furthermore, because the spring that was conventionally used has been eliminated, it is possible to make the device more compact than conventional configurations.

[0047] Furthermore, by using non-magnetic materials for the spherical members 1210a to 1210c interposed between the moving body 1200 and the base portion 1300, they will not be attracted to the magnets 1301a and 1301b when assembling the sensor module 1000, thereby improving assembly ease.

[0048] (2-2. Configuration and Control of Imaging Apparatus According to Embodiment of Present Disclosure) Next, the configuration and control of the imaging apparatus according to an embodiment of the present disclosure will be described.

[0049] (2-2-1. Configuration According to the Embodiment) First, the configuration of the imaging device according to the embodiment will be described.

[0050] 8 is a block diagram showing an example of a general configuration of an imaging device that performs video see-through according to existing technology. In FIG. 8, electronic device 500 to which the imaging device according to existing technology is applied includes imaging unit 510, imaging processing unit 520, display control unit 530, display 540, and control unit 550.

[0051] The control unit 550 includes a processor such as a CPU (Central Processing Unit) and controls, according to a program, the overall operation of the electronic device 500. The control unit 550 may also include a signal generation unit that generates a synchronization signal for operating the image sensor 512 and the imaging processing unit 520.

[0052] The imaging unit 510 includes a lens unit 511 and an image sensor 512. The image sensor 512 includes, as an imaging element, a pixel array in which pixels that output signals in response to incident light are arranged in a two-dimensional lattice pattern. The lens unit 511 collects light from the subject and irradiates the light-receiving surface of the image sensor 512. The image sensor 512 converts an analog signal (pixel signal) in response to the light received on the light-receiving surface into digital image data and outputs the digital image data to the imaging processing unit 520.

[0053] The imaging processing unit 520 includes a signal processing unit 521 and an image processing unit 522. The signal processing unit 521 performs predetermined signal processing such as demosaic processing on the pixel data output from the imaging unit 510 to generate image data on a frame-by-frame basis. The image processing unit 522 performs predetermined image processing such as color correction on the image data generated by the signal processing unit 521. The image processing unit 522 outputs the image data that has been subjected to the image processing to the display control unit 530.

[0054] The signal processing unit 521 and the image processing unit 522 may be configured by an ISP (Image Signal Processor) and a DSP (Digital Signal Processor), respectively.

[0055] The display control unit 530 generates a display signal that can be displayed on the display 540, based on the image data output from the image processing unit 522. The display 540 displays a screen according to the display signal generated by the display control unit 530 based on the image data.

[0056] In such a configuration, if the image sensor 512 has a high resolution, the amount of image data transferred per frame from the image sensor 512 to the image capture processing unit 520 increases, and there is a possibility that the data transfer will not be completed within one frame period. In such a case, in order to complete the data transfer of one frame within one frame period, it is necessary to take measures such as lowering the frame rate or thinning out the image data of one frame. However, these methods degrade the display image quality and make it impossible to take advantage of the high resolution.

[0057] 9 is a block diagram illustrating an example of the configuration of an imaging device that performs video see-through using super-resolution and is applicable to an embodiment of the present disclosure. In FIG. 9, an electronic device 1 to which an imaging device according to existing technology is applied includes an imaging unit 100, an imaging processing unit 120, a display control unit 130, a display 140, and a control unit 150. The imaging device according to the embodiment may include the imaging unit 100 and the imaging processing unit 120 of the configuration in FIG. 9.

[0058] The control unit 150 includes a processor such as a CPU and memories such as a ROM (Read Only Memory) and a RAM (Random Access Memory), and controls the overall operation of the electronic device 500 according to a program. The control unit 150 may also include a clock signal generation unit (not shown) that generates a clock signal for operating the image sensor 110 and the imaging processing unit 120. The clock signal generated by the clock signal generation unit is supplied to the imaging unit 100, the imaging processing unit 120, and the display control unit 130.

[0059] The imaging unit 100 corresponds to the sensor module 1000 described with reference to FIGS. 2 to 7, and includes an image sensor 110, a lens unit 111, a drive unit (DRV) 112, and an actuator (ACT) 113.

[0060] 2 to 7, and includes, as an imaging element, a pixel array unit in which pixels that output signals in response to incident light are arranged in a two-dimensional lattice pattern, and is fixed to the moving body 1200. The image sensor 110 generates a vertical synchronization signal Vsync and a horizontal synchronization signal Hsync based on a clock signal supplied from a clock signal generation unit, and drives the pixel array unit based on these vertical synchronization signal Vsync and horizontal synchronization signal Hsync to perform imaging (exposure).

[0061] The image sensor 110 converts analog signals (pixel signals) output from each pixel of the pixel array during image capture in response to light received on the light-receiving surface into digital pixel data and outputs the digital pixel data. The pixel data output from the image sensor 110 is supplied to the imaging processing unit 120.

[0062] The driver 112 generates a drive signal for driving the actuator 113 based on, for example, a vertical synchronization signal Vsync generated by the image sensor 110. The actuator 113 includes the piezoelectric element 1400 described with reference to Figures 2 to 7, and is driven based on the drive signal generated by the driver 112 to move the position of the image sensor 110. The image sensor 110 captures an image each time its position is moved based on the drive signal.

[0063] 8 , the imaging processing unit 120 includes a signal processing unit 121 and an image processing unit 122, which are constituted by, for example, an ISP and a DSP, respectively. The signal processing unit 121 performs predetermined signal processing such as demosaic processing on the pixel data output from the imaging unit 510 to generate image data on a frame-by-frame basis. The image processing unit 122 performs predetermined image processing such as color correction on the image data generated by the signal processing unit 121. The image processing unit 122 outputs the image data that has been subjected to the image processing to the display control unit 130.

[0064] 10 is a schematic diagram for outlining imaging control by an imaging device according to an embodiment. In FIG. 10 , an image sensor 110 captures images in accordance with a frame cycle based on a vertical synchronization signal Vsync and a horizontal synchronization signal Hsync (not shown). The image sensor 110 outputs pixel data obtained by capturing images for each frame. That is, the image sensor 110 alternately outputs pixel data of odd-numbered frames Frame (odd) and pixel data of even-numbered frames Frame (even).

[0065] Meanwhile, the driver 112 generates a drive signal based on the vertical synchronization signal Vsync generated by the image sensor 110 to drive the actuator 113, thereby shifting the position of the image sensor 110 for each frame. Here, it is assumed that the position of the image sensor 110 is shifted by a half-pixel shift amount, as described with reference to FIG. 1A . In the example of FIG. 10 , the image sensor 110 is shifted by a half-pixel in both the horizontal and vertical directions from position Posi#1 in odd-numbered frames Frame (odd) for even-numbered frames Frame (even) to capture images.

[0066] The image capturing processing unit 120 alternately outputs image data (image #A) of odd-numbered frames (odd) and image data (image #B) of even-numbered frames (even) in a frame cycle.

[0067] Returning to the explanation of Figure 9, the display control unit 130 generates a display signal for displaying on the display 140 the image data (image #A) of the odd-numbered frame Frame (odd) output from the imaging processing unit 120 and the image data (image #B) of the even-numbered frame Frame (even).

[0068] 11 is a schematic diagram illustrating a display signal according to an embodiment. As shown in the figure, the display control unit 130 outputs image data (image #A) of odd-numbered frames Frame (odd) and image data (image #B) of even-numbered frames Frame (even) alternately in a frame cycle to the display 140. Image #A based on image data of odd-numbered frames Frame (odd) and image #B based on image data of even-numbered frames Frame (even) are shifted in position by ½ pixel in both the horizontal and vertical directions.

[0069] Therefore, when images #A and #B displayed on display 140 are observed, as described using section (b) of FIG. 1B , due to the persistence of vision effect, image #A and image #B, which is shifted in position by half a pixel in both the horizontal and vertical directions relative to image #A, are observed as if they were being displayed simultaneously. Therefore, it is possible to observe a super-resolution image having a resolution twice that of the image captured by image sensor 110. The amount of image data transferred from image sensor 110 to imaging processing unit 120 is set to the amount of image data having a resolution half the observed resolution in both the horizontal and vertical directions.

[0070] FIG. 12 is a block diagram illustrating an example of a hardware configuration of an electronic device 1 to which an imaging device according to an embodiment is applied.

[0071] 12 , the electronic device 1 includes a CPU 600, a ROM 601, a RAM 602, a storage device 603, a data I / F 604, and an input device 605. The electronic device 1 further includes a lens unit 610, a sensor module 1000, an ISP 612, a DSP 613, and a display control unit 614. These units included in the electronic device 1 are connected to each other via a bus 620 so as to be able to communicate with each other.

[0072] A display 630 is connected to the display control unit 614. The display 630 may be included in the electronic device 1, or may be externally connected to the electronic device 1.

[0073] The storage device 603 is a non-volatile storage medium such as a flash memory, a hard disk drive, etc. The CPU 600 controls the overall operation of the electronic device 1 in accordance with the programs stored in the ROM 601 and the storage device 603, using the RAM 602 as a work memory.

[0074] The data I / F 604 is an interface for transmitting and receiving data to and from external devices. The data I / F 604 may transmit and receive data via a wired connection or a wireless connection. Furthermore, the data I / F 604 may communicate with a communication network such as the Internet or a LAN (Local Area Network).

[0075] The input device 605 is a device for receiving user operations, such as various switches, buttons, and a touch panel.

[0076] The lens unit 610 includes a condensing lens and may further include an autofocus mechanism, an autoiris mechanism, and a zoom mechanism. The lens unit 610 condenses light from a subject and irradiates the light onto the sensor module 1000. The sensor module 1000 outputs image data corresponding to the irradiated light to the ISP 612. The ISP 612 and DSP 613 perform processing corresponding to, for example, the signal processing unit 121 and the image processing unit 122 described above, respectively. The display control unit 614 corresponds to the display control unit 130 described above and generates a display signal that can be displayed on the display 630 based on the image data output from the DSP 613. The display 630 displays a screen corresponding to the display signal generated by the display control unit 614.

[0077] (2-2-2. Control According to the Embodiment) Next, the control according to the embodiment will be described in more detail.

[0078] 13 is a schematic diagram for explaining the effect of super-resolution applicable to the embodiment, where the image sensor 110 captures images at 2K resolution.

[0079] As shown in section (a) of FIG. 13 , image #A is captured in odd-numbered frame Frame (odd). In even-numbered frame Frame (even), the position of the image sensor 110 is shifted by ½ pixel in both the horizontal and vertical directions relative to the time of capturing odd-numbered frame Frame (odd), and image #B is captured. In this manner, images #A and #B of two odd-numbered and even-numbered frames captured at 2K resolution are displayed in chronological order. Image #B is an image whose position is shifted by ½ pixel in both the horizontal and vertical directions relative to image #A.

[0080] When a person observes a screen on which images #A and #B are displayed alternately at a frame rate, as shown in section (b) of Figure 13, images #A and #B are combined with an afterimage effect, and are recognized as a high-resolution image exceeding 2K resolution.

[0081] FIG. 14 is a timing chart illustrating an example of super-resolution image capturing and display processing that can be applied to the embodiment.

[0082] 14 , the driver 112 generates a drive signal 70 for driving the actuator 113 in synchronization with a vertical synchronization signal Vsync (not shown) and supplies the drive signal 70 to the actuator 113. When the drive signal 70 is high, for example, the actuator 113 moves the image sensor 110 to position Posi#1, and when the drive signal 70 is low, the actuator 113 moves the image sensor 110 to position Posi#2, which is shifted by ½ pixel in both the horizontal and vertical directions from position Posi#1. In the example shown in the figure, position Posi#2 is shifted downward and right on the figure from position Posi#1.

[0083] The image sensor 110 repeatedly captures image #A (or image #A') in odd-numbered frames Frame (odd) and image #B (or image #B') in even-numbered frames Frame (even) using frame timing synchronized with the vertical synchronization signal Vsync. That is, in the example of Fig. 14, the image sensor 110 captures image #A in even-numbered frames Frame (even) at a position shifted 1 / 2 pixel to the lower right in the figure relative to odd-numbered frames Frame (odd).

[0084] 14, image #A and image #B are reference images corresponding to the above-mentioned FIG. 13, and image #A' and image #B' are actually captured images. In the following, to avoid complexity, unless otherwise specified, image #A or image #A' will be represented by image #A', and image #B or image #B' will be represented by image #B'.

[0085] The image sensor 110 transfers image data (RAW data) obtained by shooting to the image capture processing unit 120. At this time, since there is a limit to the transfer rate in communication between the image sensor 110 and the image capture processing unit 120, the image sensor 110 transfers the image data of images #A' and #B' to the image capture processing unit 120 in a time-division manner. The time-division referred to here may mean transferring images #A' and #B' sequentially.

[0086] The imaging processing unit 120 performs predetermined processing such as demosaic processing and super-resolution processing on the image data of images #A' and #B' transferred from the image sensor 110. Here, in the super-resolution processing, for example, since image #B' is shifted by half a pixel from image #A', the position of each pixel in image #B' is shifted by half a pixel to correct the shift.

[0087] The image capturing processing unit 120 sequentially displays the image #A' and the image #B' with the pixel positions shifted in this way in a time-division manner on the display 140. This allows the user to observe an image with a higher resolution than the resolution of the image sensor 110 due to the afterimage effect.

[0088] FIG. 15 is a schematic diagram for explaining the configuration and processing flow of the imaging device according to the embodiment.

[0089] 15 , the drive unit 112 generates a drive signal 70 using a rectangular wave to drive the actuator 113 in accordance with timing control based on a synchronization signal 80, for example, a vertical synchronization signal Vsync, output from the sensor module 1000. The actuator 113 is driven to move the moving body 1200 in a predetermined direction in accordance with the expansion and contraction of the piezoelectric element 1400 controlled by the drive signal 70 supplied from the drive unit 112.

[0090] The sensor module 1000 may be provided with a position sensor 200 that detects the position of the moving body 1200 or the image sensor 1100 fixed to the moving body 1200. The driving unit 112 may correct the amount of movement of the moving body 1200 caused by the actuator 113 by controlling the amplitude of the driving signal 70 in accordance with the detection result of the position sensor 200.

[0091] In the sensor module 1000, the image sensor 110 captures an image each time the moving object 1200 moves in response to the drive signal 70. For example, the sensor module 1000 captures an image using light incident on the image sensor 1100 via the lens unit 1110 in response to the synchronization signal 80. The sensor module 1000 outputs image data (RAW data) obtained by capturing an image to the image capture processing unit 120. The image capture processing unit 120 performs image processing such as demosaic processing on the image data output from the sensor module 1000 and outputs the image. The display control unit 130 causes the display 140 to display an image based on the image data that has been subjected to image processing and that has been output from the image capture processing unit 120.

[0092] 16 is a schematic diagram illustrating the stability of the image sensor 110 (image sensor 1100) against positional movement according to an embodiment. In FIG. 16, the horizontal axis represents time, and the vertical axis represents position relative to a predetermined zero point. A characteristic line 50 represents the position of the image sensor 110.

[0093] 16, it is assumed that the driving signal 70 switches between high and low states at times t1, t2, t3, t4, t5, t6, ..., and the moving body 1200 moves. As shown in the figure, the position of the image sensor 110 is not stable immediately after the moving body 1200 (image sensor 110) moves. Therefore, the image sensor 110 is controlled so that images are captured during each period 51 in which the position of the image sensor 110 is stable.

[0094] 17 is a schematic diagram illustrating an example of control of the timing of capturing images for the image sensor 110 according to the embodiment. In FIG. 17, the horizontal axis represents time. The image sensor 110 starts exposure (capturing images) in synchronization with a synchronization signal 80, which is, for example, a vertical synchronization signal Vsync, and ends exposure at a predetermined timing within one frame period. The image sensor 110 reads out signals from each pixel whose exposure has been completed during the period from the end of exposure to the start of exposure for the next frame.

[0095] The drive unit 112 adjusts the rising and falling timings of the drive signal 70 by a predetermined delay time t from the rising timing of the vertical synchronization signal Vsync. sft That is, the driving unit 112 generates the driving signal 70 so that the moving body 1200 moves during a period in which exposure is not performed in the image sensor 110, and exposure starts and ends during a period 51 in which the position of the image sensor 110 is stable. More specifically, the driving unit 112 determines the timing based on the delay time t from the rising edge of the vertical synchronization signal Vsync. sft The actuator 113 moves the moving body 1200 in response to the rising and falling edges of the drive signal 70.

[0096] (2-2-3. Example of Pixel Shift Photography According to Embodiment) Next, pixel shift photography according to an embodiment will be described in more detail. In pixel shift photography according to existing technology, as described using, for example, FIGS. 1A and 1B , the image sensor is moved by equal amounts in the horizontal and vertical directions. In contrast, in an embodiment of the present disclosure, pixel shift photography is performed by different amounts of movement in the horizontal direction and the vertical direction.

[0097] 18 is a schematic diagram for explaining pixel shift photography according to an embodiment. Note that in sections (a) to (c) of Fig. 18, the right side shows the pixel shift pattern for the pixels 10 included in the pixel array 11, and the left side shows the drive signal 70. Furthermore, the pixels 10 are arranged in a Bayer array based on the color of the color filter, and of the 2 pixels x 2 pixels in the Bayer array, the upper left is an R pixel, the lower right is a B pixel, and the upper right and lower left are G pixels, respectively.

[0098] Section (a) of Figure 18 shows a first example pattern of pixel shift photography. In this first example pattern, the pixel array 11 (image sensor 1100) is shifted by one pixel horizontally and two pixels vertically, resulting in two pixel shift steps, including a state in which no pixel shift is performed. The drive signal 70a is a simple rectangular wave with signal levels that perform no shift and levels that shift the pixel array 11 diagonally by one pixel horizontally and two pixels vertically.

[0099] Section (b) of Figure 18 shows a second example pattern of pixel shift photography. In this second example pattern, a pattern in which the pixel array 11 is shifted by half a pixel horizontally and one pixel vertically is repeated three times, resulting in four steps of pixel shifting. The drive signal 70b has a signal level that changes in a stepped manner in three stages, and each of the three signal levels is a level that shifts the pixel array 11 diagonally by half a pixel horizontally and one pixel vertically.

[0100] Section (c) of Figure 18 shows a third example pattern of pixel shift photography. In this third example pattern, the pixel array 11 is shifted by a quarter pixel horizontally and a half pixel vertically, and this pattern is repeated seven times to perform eight pixel shifts. The drive signal 70c has a signal level that changes stepwise in seven steps, and each of the seven signal levels shifts the pixel array 11 by a quarter pixel horizontally and a half pixel vertically, diagonally.

[0101] In each of these first to third patterns, the amount of pixel shift is different in a first direction (e.g., the row direction) along the two-dimensional lattice in which the pixels 10 are arranged in the pixel array 11, and in a second direction (e.g., the column direction) different from the first direction, and the pixels are shifted obliquely with respect to the two-dimensional lattice. Furthermore, in each of the first to third patterns, the pixels are shifted in a direction in which the G pixels do not overlap after the shift. Furthermore, in each of the first to third patterns, the pixel array 11 is shifted to each of the divided positions obtained by dividing the period in which the original arrangement is reproduced by shifting the pixel array 11.

[0102] With reference to the first pattern example shown in section (a), for example, when the pixel array 11 shown in the first pattern example is shifted by one pixel horizontally and two pixels vertically, the relative positional relationship between the R, G, and B pixels remains unchanged before and after the shift. However, for example, after the shift, a G pixel is located in the position of the R pixel before the shift, and the absolute positional relationship of the pixel array after the shift relative to the pixel array before the shift changes, and the original array is not reproduced. On the other hand, when the pixel array 11 is shifted by two pixels horizontally and four pixels vertically, the R pixel remains in the position of the R pixel before the shift after the shift, and the absolute positional relationship of the pixel array after the shift relative to the pixel array before the shift does not change, and the original array is reproduced.

[0103] In this way, when the pixel array 11 is shifted diagonally relative to the two-dimensional lattice, the absolute positional relationship with respect to the original arrangement does not change, and the period at which the arrangement is reproduced is, for example, two pixels horizontally and four pixels vertically in a 2 pixel x 2 pixel Bayer arrangement. Therefore, in the first pattern example of section (a), the shift positions of the pixel array 11 are positions obtained by dividing this period in half. In the second pattern example of section (b), the shift positions of the pixel array 11 are positions obtained by dividing this period into four. Furthermore, in the third pattern example of section (c), the shift positions of the pixel array 11 are positions obtained by dividing this period into eight.

[0104] Next, we will explain which direction the pixels should be shifted in order to efficiently obtain the super-resolution effect. In Figures 19A and 19B, the pixel array and pixel shift direction are shown on the left, and the spatial resolution is shown on the right. In Figures 19A and 19B, the R, G, and B pixels are arranged in a Bayer array of 2 pixels x 2 pixels, with the G pixel at the top left and bottom right, the R pixel at the top right, and the B pixel at the bottom left.

[0105] 19A is a schematic diagram showing the spatial resolution of a Bayer array without pixel shifting. In this case, as shown on the right side of FIG. 19A , the spatial resolution for R and B is one unit in the vertical, horizontal, and diagonal directions, respectively, and for G is two units in the vertical and horizontal directions and one unit in each diagonal direction. Note that the dotted-line square in the outer frame is an example where there are four R, B, and G pixels, and shows the ideal spatial resolution in a 2-pixel by 2-pixel Bayer array.

[0106] Fig. 19B is a schematic diagram showing the spatial resolution of a Bayer array when pixel shifting is performed. Section (a) of Fig. 19B shows an example of pixel shifting by one pixel in the horizontal direction. In this case, the R and B colors are shifted by one unit in the vertical direction and two units in the horizontal direction, resulting in asymmetrical movement between the vertical and horizontal directions. On the other hand, the G color is shifted by two units in each of the vertical and horizontal directions, resulting in ideal spatial resolution.

[0107] Section (c) of Fig. 19B shows an example in which a pixel shift is performed by one pixel in a 45° diagonal direction, in other words, in the direction of the G pixel arrangement. In this case, the R and B colors are shifted by two units in the vertical and horizontal directions, and one unit in the diagonal direction. The G color is the same as the example in Fig. 19A in which no pixel shift is performed.

[0108] Section (d) of Fig. 19B shows an example of pixel shifting in four directions by one pixel at a time. In this case, the R, B, and G colors are all shifted by two units each in the vertical and horizontal directions, resulting in ideal spatial resolution.

[0109] Of these, the example of pixel shifting in four directions shown in section (c) of Figure 19B provides the highest spatial resolution, but the mechanism for achieving pixel shifting is complex. On the other hand, in the examples of pixel shifting in one direction shown in sections (a) and (b), the example of section (a) is asymmetric in the vertical and horizontal directions, while the example of section (b) is balanced in each direction. From the above, it is considered preferable to shift pixels in a diagonal direction.

[0110] (2-2-4. Specific Example of Pixel Shift Photography According to Embodiment) Next, pixel shift photography according to an embodiment of the present disclosure will be described more specifically.

[0111] 20A to 20C are schematic diagrams illustrating pixel shift photography according to an embodiment. Fig. 20A shows an example of a Bayer array without pixel shifting. Fig. 20A is similar to Fig. 19A described above and is provided for comparison.

[0112] Fig. 20B shows a first example pattern of pixel shift photography described in section (a) of Fig. 18, in which the pixel array 11 (image sensor 1100) is shifted by one pixel in the horizontal direction and two pixels in the vertical direction, as shown in section (b). Also, as shown in section (a), the drive signal 70a is a simple square wave whose signal level changes in two stages, similar to the example in section (a) of Fig. 18.

[0113] Section (c) of Figure 20B shows an example of pixel shift photography using the first pattern example shown in section (b), with each pixel 10 focusing on an R pixel, a B pixel, and a G pixel, overlaid as a 4 x 4 pixel block. The diagram on the left side of section (c) shows an example focusing on R pixels, with the first and third rows all being R. The diagram in the center of section (c) shows an example focusing on B pixels, with the second and fourth rows all being B pixels. The diagram on the right side of section (c) shows an example focusing on G pixels, with all 4 x 4 pixels being G pixels.

[0114] Section (d) of Figure 20B shows the spatial resolution for this first example pattern. In this case, the R and B colors are one unit vertically and two units horizontally, resulting in vertical and horizontal asymmetric resolution. Meanwhile, the G color is two units vertically and two units horizontally, resulting in ideal spatial resolution.

[0115] 20C is an example of pixel shift photography in which the pixel array 11 is shifted by one pixel in the horizontal direction, as shown in section (b). As shown in section (a), the drive signal 70a′ is a simple square wave whose signal level shifts the pixel array 11 by one pixel in the horizontal direction.

[0116] Section (c) of Figure 20C shows an example of pixel shift photography using the pixels 10 shown in section (b) shifted by one pixel horizontally, with the R, B, and G pixels being focused on and overlapped as 4 x 4 pixel blocks. This is similar to the first pattern example described above. That is, in the example focusing on the R pixels on the left side of section (c), the first and third rows are all R, while in the example focusing on the B pixels in the center of section (c), the second and fourth rows are all B pixels. Furthermore, in the example focusing on the G pixels on the right side of section (c), all 4 x 4 pixels are G pixels.

[0117] Section (d) of Figure 20C shows the spatial resolution when pixel shift photography is performed with a horizontal shift of one pixel, as shown in section (b). In this case, the R and B colors have a vertical resolution of one unit and a horizontal resolution of two units, and the G color has a vertical resolution of two units and a horizontal resolution of two units, which is ideal. This spatial resolution is equivalent to section (d) of Figure 20B.

[0118] As can be seen by comparing section (d) of Fig. 20B with section (d) of Fig. 20C, there is no difference in spatial resolution between pixel shift photography performed with a horizontal shift of one pixel and a vertical shift of two pixels and pixel shift photography performed with a horizontal shift of only one pixel. However, pixel shift photography using the first example pattern shown in Fig. 20B, in which a horizontal shift of one pixel and a vertical shift of two pixels is performed, is extensible.

[0119] Fig. 21 is a schematic diagram showing an extended example of pixel shift photography using the first pattern example according to an embodiment. The example of Fig. 21 corresponds to the second pattern example of pixel shift photography described in section (b) of Fig. 18 , and a pattern in which the pixel array 11 is shifted by half a pixel in the horizontal direction and one pixel in the vertical direction is repeated three times. The drive signal 70b is a signal whose signal level changes in a stepped manner, and each of the three signal levels is a level that shifts the pixel array 11 diagonally by half a pixel in the horizontal direction and one pixel in the vertical direction.

[0120] Section (c) of Figure 21 shows an example in which the pixel array 11 shown in section (b) is shifted by half a pixel horizontally and one pixel vertically three times, with each pixel 10 being superimposed as a 4 x 4 pixel block, focusing on R, B, and G pixels. The diagram on the left side of section (c) shows an example focusing on R pixels, with all 4 x 4 pixels being R color. The diagram in the center of section (c) shows an example focusing on B pixels, with all 4 x 4 pixels being B color. The diagram on the right side of section (c) shows an example focusing on G pixels, with all 4 x 4 pixels being G pixels.

[0121] Section (d) of Figure 21 shows the spatial resolution of this second example pattern, where the R, B, and G colors are each 2 units vertically and 4 units horizontally, resulting in higher spatial resolution in both the horizontal and vertical directions compared to the spatial resolution of the first example pattern shown in section (b) of Figure 20B.

[0122] FIG. 22 is a schematic diagram for explaining an example in which the number of steps of change in the drive signal 70 is increased according to the embodiment.

[0123] In Fig. 22, section (a) shows an example in which pixel shifting is not performed on each pixel 10 arranged in a Bayer array. Although not shown, in this example, pixel shifting is not performed, and therefore the drive signal 70 does not change. This state is referred to as a one-step change. As described with reference to Fig. 19A, the spatial resolution for R and B is one unit each in the vertical, horizontal, and diagonal directions, while the spatial resolution for G is two units each in the vertical and horizontal directions and one unit each in the diagonal directions.

[0124] Section (b) of Fig. 22 shows a first example pattern of pixel shift photography, as described in section (a) of Fig. 18 , in which pixel shifting is performed in two steps using a drive signal 70a that is a simple square wave, with signal levels of one that does not shift the pixel array 11 and one that shifts the pixel array 11 diagonally by one pixel horizontally and two pixels vertically. As described with reference to Fig. 20B , the spatial resolution for R and B is one unit vertically and two units horizontally, which is twice as high in each direction as in the example of section (a). Meanwhile, the spatial resolution for G is two units vertically and two units horizontally, which is ideal.

[0125] Section (c) of Fig. 22 shows a second example pattern of pixel shift photography described in section (b) of Fig. 18, in which pixel shifting is performed in four steps by a drive signal 70b in which the signal level changes stepwise in three stages, and each of the three signal levels is a level that diagonally shifts the pixel array 11 by half a pixel in the horizontal direction and one pixel in the vertical direction. As described with reference to Fig. 21, the spatial resolution for R, B, and G is two units vertically and four units horizontally, and compared to the example of section (b), the R and B colors are doubled in the vertical and horizontal directions, and the G color is doubled in the horizontal direction.

[0126] Section (d) of Fig. 22 shows a third example pattern of pixel shift photography described in section (c) of Fig. 18, in which pixel shifting is performed in eight steps by a drive signal 70c in which the signal level changes stepwise in seven steps, and each of the seven signal levels is a level that shifts the pixel array 11 diagonally by 1 / 4 pixel in the horizontal direction and 1 / 2 pixel in the vertical direction. Although not shown, the spatial resolution is doubled in both the vertical and horizontal directions for R, G, and B colors compared to the spatial resolution in section (c).

[0127] 22 is a pattern example (referred to as a fourth pattern example) in which the number of pixel shift steps is doubled compared to the example in section (d), and pixel shifting is performed in 16 steps by a drive signal 70d in which the signal level changes stepwise in 15 steps, and each of the 15 signal levels is a level that moves the pixel array 11 diagonally by 1 / 8 pixel in the horizontal direction and 1 / 4 pixel in the vertical direction. Although not shown, the spatial resolution is doubled in both the vertical and horizontal directions for R, G, and B colors compared to the spatial resolution in section (d).

[0128] As can be seen from sections (a) to (e) of Figure 22, even if the pixel shift is only in one axis direction, it is possible to obtain the effect of super-resolution in the vertical and horizontal directions by increasing the number of pixel shift steps.

[0129] Fig. 23 is a schematic diagram illustrating the signal level of the drive signal 70 according to the number of pixel shift steps according to an embodiment. In Fig. 23, sections (a) to (e) correspond to sections (a) to (e) in Fig. 22, respectively. A description of each pixel shift below sections (a) to (e) will be omitted.

[0130] 23, the upper diagram shows the signal level of the drive signal 70. As shown in section (a), the signal level is divided into four stages from 0 to level Lv4, with level Lv2 being the signal level that linearly moves the pixel array 11 (image sensor 1100) by a distance a, which shifts the pixel array 11 horizontally by one pixel and vertically by two pixels, and level Lv4 being the signal level that linearly moves the pixel array 11 by a distance b, which shifts the pixel array 11 horizontally by four pixels.

[0131] Section (b) of Fig. 23 shows a two-step pixel shift of one pixel horizontally and two pixels vertically according to a first example pattern of pixel shifting. In this case, the drive signal 70a is a simple square wave whose signal level is level 0, which does not perform a shift, and level Lv2, which diagonally shifts the pixel array 11 by one pixel horizontally and two pixels vertically.

[0132] Section (c) of Fig. 23 shows a four-step pixel shift in accordance with a second example pattern of pixel shifting, in which the pixel array 11 is shifted by half a pixel in the horizontal direction and by one pixel in the vertical direction, and this pattern is repeated three times. In this case, the drive signal 70b is a signal whose signal level changes in a stepped manner in three stages, and each of the three signal levels is Lv1, which is half the level Lv2.

[0133] Section (d) of Fig. 23 shows an eight-step pixel shift in accordance with a third example pattern of pixel shifting, in which the pixel array 11 is shifted by a quarter pixel in the horizontal direction and a half pixel in the vertical direction, repeated seven times. In this case, the drive signal 70c has a signal level that changes stepwise in seven steps, and each of the seven signal levels is half the level Lv1.

[0134] Section (e) of Fig. 23 shows a 16-step pixel shift in which a pattern of shifting the pixel array 11 horizontally by 1 / 8 pixel and vertically by 1 / 4 pixel is repeated 15 times according to the fourth example pattern of pixel shifting (see section (e) of Fig. 22). In this case, the drive signal 70d is a signal whose signal level changes in a stepped manner in 15 steps, and each of the 15 signal levels is 1 / 4 of the level Lv1.

[0135] Although the signal levels of the drive signals 70b to 70d shown in sections (c) to (e) of Fig. 23 change in a sawtooth waveform, the change in signal level is not limited to this example and the signal level may also be changed in a triangular waveform, for example. Fig. 24A and Fig. 24B are schematic diagrams showing an example in which the signal level of the drive signal 70 changes in a triangular waveform according to an embodiment.

[0136] Fig. 24A shows an example of changing the signal level of the drive signal 70 in the second example pattern of pixel shifting described using section (c) of Fig. 23. Section (a) of Fig. 24A shows an example of a drive signal 70b in which the signal level is changed in three stages in a sawtooth waveform, similar to section (c) of Fig. 23.

[0137] Section (b) of Figure 24A shows an example of a drive signal 70b' that changes the signal level in three stages in a triangular wave pattern. Specifically, the drive signal 70b' divides the period (in this example, a period of two pixels horizontally and four pixels vertically) at which the original arrangement is reproduced by pixel shifting of the pixel array 11 into four parts, thinning out each of the divided positions one by one up to the position with the largest amount of movement. Furthermore, the signal level is changed corresponding to each of the thinned positions among the divided positions, from the position with the largest amount of movement toward the position with the smallest amount of movement.

[0138] Fig. 24B shows an example of changing the signal level of the drive signal 70 in the third example pattern of pixel shifting described using section (d) of Fig. 23. Section (a) of Fig. 24B shows an example of a drive signal 70c in which the signal level is changed in 15 steps in a sawtooth waveform, similar to section (d) of Fig. 23.

[0139] Section (b) of Figure 24A shows an example of a drive signal 70c' that changes the signal level in a triangular wave pattern over 15 stages. Similar to the drive signal 70b' described above, the drive signal 70c' specifically changes the signal level by thinning out each of the 16 divided positions, one by one, from the position with the greatest amount of movement to the position with the greatest amount of movement among the divided positions obtained by dividing the period in which the original arrangement is reproduced by pixel shifting of the pixel array 11. Furthermore, the signal level is changed in accordance with each thinned-out position among the divided positions, from the position with the greatest amount of movement to the position with the smallest amount of movement among the divided positions.

[0140] The actuator 113 (piezoelectric element 1400) is easier to configure when driven by drive signals 70b' and 70c' whose signal level changes in a triangular wave shape with fewer harmonic components than when driven by drive signals 70b and 70c whose signal level changes in a sawtooth wave shape.

[0141] Patent Document 1 discloses a structure in which a drive sliding part has a shaft and a hole and is held down by a coil spring as a means for achieving pixel shift. However, with the configuration of Patent Document 1, the holding location is limited, making it difficult to stabilize the posture, and play occurs due to the gap between the shaft and the hole. As a result, the positional relationship between the lens and the sensor is not as it should be, which may reduce the effectiveness of improving image quality.

[0142] In contrast, according to the technology of the present disclosure, by supporting the V-groove 1202 and the surface of the base portion 1300 facing the mobile object 1200 with three spherical members 1210a to 1210c, the mobile object 1200 can be held without any gaps, and by arranging the three spherical members 1210a to 1210c that serve as support portions at a distance from each other, the center of gravity of the mobile object 1200 can be located inside, making it possible to stabilize its posture. Furthermore, according to the technology of the present disclosure, by using the magnetic force of a permanent magnet as a holding force, it is possible to simplify the structure of the sensor module 1000.

[0143] Therefore, by applying the technology of the present disclosure, it is possible to improve the super-resolution effect by achieving posture stability that suppresses rattles in the pixel shift drive. Also, because the magnetic force of a permanent magnet is used to hold down the moving body 1200, the sensor module 1000 can be made smaller, which improves the degree of freedom in placement in the product, improves durability, and extends the product life.

[0144] Furthermore, Patent Document 2 describes examples of drive directions, such as a 45° direction relative to the image sensor pixel array, or the horizontal and vertical directions, but does not describe other drive directions. Furthermore, regarding the super-resolution effect, because the arrangement of each RGB pixel is different in the Bayer array, the effect differs for each RGB color depending on the drive direction and method.

[0145] In contrast, according to the technology disclosed herein, when shifting one pixel in a Bayer array, the pixel shift is performed so that the positions of the G pixels do not overlap before and after the pixel shift, thereby making it possible to increase the resolution in one direction for each of the RGB colors.

[0146] Furthermore, the pixel shift of one pixel in the horizontal direction and two pixels in the vertical direction according to the technology disclosed herein has the same super-image effect as the pixel shift of one pixel in the horizontal direction, but has the advantage of being more scalable. By performing pixel shifting using a drive signal whose signal level changes stepwise, it is possible to expand the spatial resolution bandwidth in the vertical direction as well. Furthermore, by increasing the number of signal level changes and the number of pixel shift steps, it is possible to obtain super-resolution effects in both the horizontal and vertical directions.

[0147] (3. Other Embodiments) In the above-described embodiment, pixel shifting is performed by moving the image sensor 110 (image capture element 1100) to obtain super-resolution, but this is not limited to this example. For example, pixel shifting may be achieved by moving the lens unit 1110 relative to the image capture element 1100.

[0148] The effects described in this specification are merely examples and are not limiting, and other effects may also be present.

[0149] The present technology can also be configured as follows. (1) An imaging device comprising: a movement mechanism that moves an imaging element including a pixel array in which pixels that generate signals in response to incident light are arranged according to a two-dimensional lattice and optical filters with different characteristics are provided for each of the pixels in a predetermined arrangement, on a plane perpendicular to the incident direction of the light, in a first direction along the two-dimensional lattice and a second direction different from the first direction, and obliquely relative to the two-dimensional lattice; and a control unit that drives and controls the movement mechanism and controls imaging of the imaging element, wherein the control unit performs imaging using the imaging element each time the imaging element is moved by the movement mechanism. (2) The imaging device described in (1), wherein the control unit drives and controls the movement mechanism to move the imaging element to each of divided positions obtained by dividing a period in which the predetermined arrangement is reproduced by movement by the movement mechanism. (3) The imaging device described in (2), wherein the divided positions are positions obtained by dividing the period by a power of two. (4) The imaging device according to (2) or (3), wherein the control unit drives and controls the movement mechanism to sequentially move the imaging element to each of the division positions. (5) The imaging device according to (2) or (3), wherein the control unit drives and controls the movement mechanism to sequentially move each of the division positions up to the position among the division positions with the largest movement amount, while thinning out each of the division positions, and sequentially move each of the thinned division positions from the position with the largest movement amount toward the position among the division positions with the smallest movement amount. (6) The imaging device according to any one of (1) to (5), wherein the control unit moves the imaging element in synchronization with a frame period of imaging by the imaging element. (7) The imaging device according to any one of (1) to (6), wherein the movement mechanism moves the imaging element using a piezoelectric element that expands and contracts in one direction in response to voltage. (8) The imaging device according to (7), wherein the moving mechanism includes a base portion to which the piezoelectric element is fixed, and a moving body provided in contact with an end portion of the piezoelectric element in the expansion / contraction direction and moved relative to the base portion in accordance with the expansion / contraction of the piezoelectric element, and the imaging element is fixed to the moving body.(9) The imaging device according to (8), wherein the base portion is provided with a first magnet whose magnetic force direction is aligned with the expansion and contraction direction of the piezoelectric element, and the moving body is provided with a first magnetic body facing the first magnet. (10) The imaging device according to (8) or (9), wherein the moving mechanism further includes a plurality of spheres sandwiched between a first surface of the base portion on which the piezoelectric element is provided and a second surface of the moving body facing the first surface and parallel to the first surface. (11) The imaging device according to (10), wherein the movable body has, on the second surface, a V-shaped groove extending in a direction of movement relative to the base portion, and one recess; the base portion includes, on the first surface, two holes corresponding to the V-shaped groove and one hole corresponding to the one recess; each of the holes formed on the first surface has a conical surface with an opening on the first surface as a bottom surface; at least two of the plurality of true spheres are sandwiched between the V-shaped groove and the plurality of holes formed corresponding to the V-shaped groove; and at least one of the plurality of true spheres other than the one sandwiched between the V-shaped groove and the plurality of holes is sandwiched between the one recess and the one hole formed corresponding to the one recess. (12) The imaging device according to (11), wherein the two holes and the one hole are arranged in a positional relationship such that a center of gravity of the movable body and a driving range for moving the movable body are included inside a triangle formed by the two holes and the one hole. (13) The imaging device according to any one of (10) to (12), wherein the base portion is provided with a second magnet whose magnetic force direction is aligned from the first surface to the second surface, and the movable body is provided with a second magnetic material on the second surface. (14) The imaging device according to any one of (10) to (13), wherein each of the plurality of spheres is a non-magnetic material. (15) The imaging device according to any one of (10) to (14), wherein each of the plurality of spheres is ceramic. (16) The imaging device according to any one of (1) to (15), wherein the predetermined arrangement is a Bayer arrangement.(17) An electronic device comprising: a movement mechanism that moves an imaging element including a pixel array in which pixels that output signals in response to incident light are arranged according to a two-dimensional lattice and optical filters with mutually different characteristics are provided for each of the pixels in a predetermined arrangement, on a plane that intersects perpendicularly with the incident direction of the light, in a first direction along the two-dimensional lattice and a second direction different from the first direction, and obliquely relative to the two-dimensional lattice; a control unit that performs drive control of the movement mechanism and image capture control of the imaging element; and an image capture processing unit that performs signal processing on the signal output from the imaging element to generate image data, wherein the control unit performs image capture by the imaging element each time the imaging element is moved by the movement mechanism.

[0150] 1 Electronic device 10 Pixel 11 Pixel array 20, 20sft Display pixel 70, 70a, 70b, 70c, 70d Drive signal 80 Synchronization signal 100, 510 Imaging unit 110, 512 Image sensor 111, 511 Lens unit 112 Drive unit 113 Actuator 120, 520 Imaging processing unit 121, 521 Signal processing unit 122, 522 Image processing unit 130, 530 Display control unit 140, 540 Display 150, 550 Control unit 1000 Sensor module 1100 Imaging element 1110 Lens unit 1200 Moving body 1300 Base unit 1301a, 1301b Magnet 1302a, 1302b Yoke 1201a, 1201b, 1201c Panel 1202 V-groove 1203 Recess 1210a, 1210b, 1210c Spherical member 1220 Protrusion 1303a, 1303b, 1303c Hole 1304 Conical surface 1305, 1306 Magnetic force 1400 Piezoelectric element

Claims

1. An imaging device comprising: a movement mechanism that moves an imaging element including a pixel array in which pixels that generate signals in response to incident light are arranged according to a two-dimensional lattice, and in which optical filters with different characteristics are provided for each of the pixels in a predetermined arrangement, on a plane that intersects perpendicularly with the direction of incidence of the light, in a first direction along the two-dimensional lattice and a second direction different from the first direction, and obliquely relative to the two-dimensional lattice; and a control unit that controls the drive of the movement mechanism and controls the imaging of the imaging element, wherein the control unit performs imaging using the imaging element each time the imaging element is moved by the movement mechanism.

2. The imaging device according to claim 1, wherein the control unit controls the drive of the moving mechanism to move the imaging element to each of the divided positions obtained by dividing the period in which the predetermined arrangement is reproduced by movement by the moving mechanism.

3. The imaging device according to claim 2, wherein the division positions are positions obtained by dividing the period by a power of two.

4. The imaging device according to claim 2, wherein the control unit drives and controls the movement mechanism to move the imaging element sequentially to each of the division positions.

5. The imaging device of claim 2, wherein the control unit drives and controls the movement mechanism to move the division positions sequentially, thinning out each of the division positions, up to the position among the division positions with the largest movement amount, and then moves the division positions sequentially, one by one, from the position with the largest movement amount toward the position among the division positions with the smallest movement amount.

6. The imaging device according to claim 1, wherein the control unit moves the imaging element in synchronization with a frame period of imaging by the imaging element.

7. The imaging device according to claim 1, wherein the movement mechanism moves the imaging element using a piezoelectric element that expands and contracts in one direction in response to voltage.

8. The imaging device described in claim 7, wherein the moving mechanism includes a base portion to which the piezoelectric element is fixed, and a movable body provided in contact with an end portion of the piezoelectric element in the expansion / contraction direction and moved relative to the base portion in accordance with the expansion / contraction of the piezoelectric element, and the imaging element is fixed to the movable body.

9. The imaging device according to claim 8, wherein the base portion is provided with a first magnet whose magnetic force direction is aligned with the direction of expansion and contraction of the piezoelectric element, and the movable body is provided with a first magnetic body facing the first magnet.

10. The imaging device described in claim 8, wherein the moving mechanism further includes a plurality of spheres sandwiched between a first surface of the base portion on which the piezoelectric element is provided and a second surface of the moving body that faces the first surface and is parallel to the first surface.

11. The imaging device described in claim 10, wherein the movable body has, on the second surface, a V-shaped groove extending in the direction of movement relative to the base portion, and one recess; the base portion includes, on the first surface, two holes corresponding to the V-shaped grooves and one hole corresponding to the one recess; each of the holes formed on the first surface has a conical surface with an opening in the first surface as a bottom surface; at least two of the plurality of spheres are sandwiched between the V-shaped groove and the plurality of holes formed corresponding to the V-shaped groove; and at least one of the plurality of spheres other than the sphere sandwiched between the V-shaped groove and the plurality of holes is sandwiched between the one recess and the one hole formed corresponding to the one recess.

12. The imaging device described in claim 11, wherein the two holes and the one hole are positioned such that the center of gravity of the moving body and the driving range for moving the moving body are included within a triangle formed by the two holes and the one hole.

13. The imaging device described in claim 10, wherein the base portion is provided with a second magnet whose magnetic force direction is aligned from the first surface to the second surface, and the movable body is provided with a second magnetic material on the second surface.

14. The imaging device according to claim 10, wherein each of the plurality of spheres is made of a non-magnetic material.

15. The imaging device according to claim 10, wherein each of the plurality of spheres is made of ceramic.

16. The imaging device according to claim 1, wherein the predetermined arrangement is a Bayer arrangement.

17. An electronic device comprising: a movement mechanism that moves an imaging element including a pixel array in which pixels that output signals in response to incident light are arranged according to a two-dimensional lattice and optical filters with different properties are provided for each of the pixels in a predetermined arrangement, on a plane that intersects perpendicularly with the direction of incidence of the light, in a first direction along the two-dimensional lattice and a second direction different from the first direction, and obliquely relative to the two-dimensional lattice; a control unit that controls the drive of the movement mechanism and controls imaging of the imaging element; and an imaging processing unit that performs signal processing on the signal output from the imaging element to generate image data, wherein the control unit performs image capturing using the imaging element each time the imaging element is moved by the movement mechanism.

Images