Imaging device and electronic device

Abstract

An imaging apparatus according to the present disclosure includes: a moving mechanism in which pixels that generate a signal according to incident light are arranged so as to conform to a two-dimensional lattice, an imaging element including a pixel array in which a light filter having a different characteristic from each other with respect to each of the pixels is disposed in a predetermined array, the imaging element moving different amounts in each of a first direction along the two-dimensional lattice and a second direction different from the first direction on a plane that perpendicularly intersects with an incident direction of light, and the imaging element moving obliquely with respect to the two-dimensional lattice; and a control unit for performing drive control of the moving mechanism and imaging control of the imaging element. The control unit causes the imaging element to perform imaging each time the imaging element is moved by the moving mechanism.

Description

Technical Field

[0001] This disclosure relates to imaging devices and electronic devices. Background Technology

[0002] A technique known as video see-through (penetration) is used for real-time display of video captured by a camera device. In recent years, especially in the field of head-mounted displays (HMDs), more and more HMDs are incorporating this technology.

[0003] Reference List

[0004] Patent documents

[0005] Patent Document 1: WO 2019 / 155809

[0006] Patent Document 2: JP 2021-40261 A Summary of the Invention

[0007] Technical issues

[0008] For example, in an HMD that incorporates video perspective technology, in order to obtain a highly realistic display video, it is necessary to capture the video at high resolution and display the high-resolution captured video without delay.

[0009] Therefore, the purpose of this disclosure is to provide an imaging apparatus and electronic device that can capture video data at high resolution and transmit the video data captured at high resolution at a higher speed.

[0010] Solution to the problem

[0011] The imaging apparatus according to this disclosure includes: a moving mechanism configured to move an imaging element at an angle relative to a two-dimensional dot matrix on a plane perpendicular to the incident direction of incident light, such that the amount of movement in a first direction along the two-dimensional dot matrix is ​​different from the amount of movement in a second direction different from the first direction; the imaging element includes a pixel array in which pixels configured to generate signals according to incident light are arranged in a two-dimensional dot matrix, each pixel in the predetermined arrangement being provided with a filter with different characteristics; and a control unit configured to perform drive control of the moving mechanism and imaging control of the imaging element, wherein the control unit performs imaging using the imaging element whenever the imaging element is moved by the moving mechanism. Attached Figure Description

[0012] Figure 1A It is a schematic diagram used to describe super-resolution according to the prior art.

[0013] Figure 1B It is a schematic diagram used to describe super-resolution according to the prior art.

[0014] Figure 2 This is a schematic diagram illustrating an example of the appearance of a sensor module applied to an imaging apparatus according to an embodiment of this disclosure.

[0015] Figure 3 This is a perspective view showing an example of the structure of a sensor module according to an embodiment in an exploded manner.

[0016] Figure 4 This is a schematic diagram showing the state in which a perfectly spherical member, according to an embodiment, is sandwiched between a moving body and a base.

[0017] Figure 5 This is a schematic diagram that more specifically illustrates an example configuration of the moving body and base according to an embodiment.

[0018] Figure 6 This is a schematic diagram that more specifically illustrates an example configuration of the moving body and base according to an embodiment.

[0019] Figure 7 This is a schematic diagram used to describe the movement of a moving body according to an embodiment.

[0020] Figure 8 This is an exemplary block diagram illustrating a general configuration of an imaging apparatus for performing video perspective according to the prior art.

[0021] Figure 9 This is an exemplary block diagram illustrating the configuration of an imaging apparatus that performs video perspective using super-resolution, applicable to embodiments of this disclosure.

[0022] Figure 10 This is a schematic diagram illustrating the imaging control performed by the imaging apparatus according to an embodiment.

[0023] Figure 11 This is a schematic diagram used to describe the display signals according to the implementation method.

[0024] Figure 12 This is an exemplary block diagram illustrating the hardware configuration of an electronic device to which an imaging apparatus according to an embodiment is applied.

[0025] Figure 13 This is a schematic diagram used to illustrate the effects of super-resolution that can be applied to the implementation methods.

[0026] Figure 14 This is an exemplary timing diagram used to describe the processing of super-resolution capture and display that can be applied to implementation methods.

[0027] Figure 15 It is a schematic diagram used to describe the configuration and processing flow of the imaging apparatus according to the embodiments.

[0028] Figure 16 This is a schematic diagram used to illustrate the stability of positional movement of an image sensor (imaging element) according to an embodiment.

[0029] Figure 17 This is a schematic diagram illustrating an example of controlling the capture timing of an image sensor according to an embodiment.

[0030] Figure 18 This is a schematic diagram used to describe pixel shift capture according to an embodiment.

[0031] Figure 19A This is a schematic diagram illustrating the spatial resolution obtained using the Bayer arrangement without performing pixel shifting.

[0032] Figure 19B This is a schematic diagram illustrating the spatial resolution obtained using the Bayer arrangement when pixel shifting is performed.

[0033] Figure 20A This is a schematic diagram used to describe pixel shift capture according to an embodiment.

[0034] Figure 20B This is a schematic diagram used to describe pixel shift capture according to an embodiment.

[0035] Figure 20C This is a schematic diagram used to describe pixel shift capture according to an embodiment.

[0036] Figure 21 This is a schematic diagram illustrating an extended example of pixel shift capture according to a first mode example of an implementation.

[0037] Figure 22 This is a schematic diagram illustrating an example of increasing the number of steps in a change of the drive signal according to an implementation method.

[0038] Figure 23 This is a schematic diagram illustrating the signal level of the drive signal 70 according to the number of pixel shift steps in the embodiment.

[0039] Figure 24A This is a schematic diagram illustrating an example according to an embodiment, wherein the signal level of the drive signal is changed in a triangular waveform.

[0040] Figure 24B This is a schematic diagram illustrating an example according to an embodiment, wherein the signal level of the drive signal is changed in a triangular waveform. Detailed Implementation

[0041] In the following, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. Note that in the following embodiments, the same reference numerals denote the same parts, and redundant descriptions will be omitted.

[0042] In the following text, embodiments of the present disclosure will be described in the following order.

[0043] 1. Prior art related to the technology disclosed herein.

[0044] 2. Implementation of the Contents of this Disclosure

[0045] 2-1. Structure of an imaging apparatus according to an embodiment of the present disclosure

[0046] 2-2. Configuration and control of the imaging apparatus according to embodiments of the present disclosure

[0047] 2-2-1. Configuration according to the implementation method

[0048] 2-2-2. Control according to the implementation method

[0049] 2-2-3. Example of pixel shift capture according to the implementation method

[0050] 2-2-4. Specific example of pixel shift capture according to the implementation method

[0051] 3. Other implementation methods

[0052] 1. Prior art related to the technology disclosed herein.

[0053] First, before describing the technology based on this disclosure, prior art related to the technology of this disclosure will be described.

[0054] A technique known as video see-through (penetration) is used for real-time display of video captured by a camera device. In recent years, particularly in the field of head-mounted displays (HMDs), an increasing number of HMDs are incorporating video see-through technology. In video see-through technology, to obtain highly realistic video, it is necessary to display images captured at high resolution without delay.

[0055] When images are captured by a high-resolution camera, the processor's capacity to process the image information becomes a bottleneck, making it difficult to transmit image data. For example, even when a camera (image sensor) captures images at 4K resolution (e.g., 3840 horizontal pixels × 2160 vertical pixels) and a display capable of showing images at 4K resolution exists, it is still impossible to display moving images at 4K resolution without delay if the image information can only be transmitted at a frame rate of up to 2K resolution (e.g., 1920 horizontal pixels × 1080 vertical pixels).

[0056] Furthermore, when the imaging element supports 4K resolution and the number of pixels increases, there is a problem of reduced pixel pitch and decreased sensitivity. Although there are methods to improve sensitivity by pixel binning when the number of pixels is large, there is a problem that the inter-frame loss increases as the pixel pitch narrows, thus reducing efficiency.

[0057] On the other hand, a technique known as super-resolution is known. In super-resolution, image information is acquired separately for each of a plurality of pixel combinations of an image sensor mounted in a camera device, and these plurality of pixel combinations are shifted by one pixel. A high-resolution image is then generated by subsequently synthesizing the image information. The synthesis of image information is typically performed by software processing, and suffers from a lack of real-time performance due to time lag relative to the frame rate.

[0058] Furthermore, a known technique involves dividing an image captured at 4K resolution into multiple images, which are then displayed alternately. Even with this technique, there is a time requirement to perform the segmentation process for the 4K-captured image, resulting in a loss of real-time performance.

[0059] As a solution to these problems, a method has been proposed in which images are captured as pixels of an image sensor mounted in a camera device are physically shifted for each frame. In this method, the amount of information is reduced by dividing the image into two sets of images captured at positions that are alternately shifted for each frame, and image information is transmitted alternately for each acquired frame. For each frame, the transmitted pixel information is alternately output to a display device. Due to the persistence of vision, the displayed image can be observed as a high-resolution image.

[0060] Figure 1A and Figure 1B This is a schematic diagram used to describe super-resolution based on existing technology. Figure 1A and Figure 1BIn the example, pixel shift capture is performed on the image sensor pixels for each frame, and a set of frames obtained by pixel shift capture is displayed alternately during display, so as to obtain a super-resolution display image.

[0061] Figure 1A This illustrates the processing involved in capturing an image using pixel-shift capture. Figure 1A As shown in section (a), in the pixel array 11 included in the image sensor, pixels 10 are arranged in a two-dimensional dot matrix pattern. The image sensor captures image information at position Posi#1 in each odd-numbered frame (odd frame), and captures image information at position Posi#2 shifted by 1 / 2 pixel in each of the horizontal and vertical directions in each even-numbered frame (even frame).

[0062] As shown in part (b), for example, the image sensor transmits image information captured at location Posi#1 and image information captured at location Posi#2, respectively.

[0063] Figure 1B The processing during display is illustrated. During display, as shown in section (a), for each frame, image 21 formed by display pixels 20 based on image information captured at position Posi# and image 21sft formed by display pixels 20sft based on image information captured at position Posi#2 are displayed alternately. At this time, if the display position shifts according to positions Posi#1 and Posi#2, image 21 and image 21sft are displayed alternately for each frame.

[0064] Display control is performed in this way, such as Figure 1B As shown in part (b), due to the persistence of vision, the image 21 formed by display pixel 20 and the image 21sft formed by display pixel 20sft are observed as if these images were displayed simultaneously. Therefore, a super-resolution image, i.e., an image with twice the resolution when captured by the image sensor, can be observed.

[0065] In embodiments of this disclosure, in a method for obtaining a super-resolution image by capturing an image while physically shifting pixels of an image sensor for each frame, the direction of pixel shifting is designed, and thus higher resolution can be achieved. As a method for shifting pixels of the image sensor, the imaging element is moved using a piezoelectric element, and the movement of the imaging element is controlled by using magnetic force.

[0066] 2. Implementation of the Contents of this Disclosure

[0067] The implementation of this disclosure will now be described.

[0068] 2-1. Structure of an imaging apparatus according to an embodiment of the present disclosure

[0069] First, the structure of an imaging apparatus according to an embodiment of the present disclosure will be described. Figure 2 This is a schematic diagram illustrating an example of the appearance of a sensor module applied to an imaging apparatus according to an embodiment of this disclosure.

[0070] exist Figure 2 Part (a) is a perspective view showing an example of the appearance of the sensor module 1000 including the lens unit 1110. In parts (a) and (b) described below, a portion of the sensor module 1000 is shown in a semi-transparent manner for ease of description.

[0071] In the sensor module 1000, a movable body 1200 is disposed above the base 1300, and an imaging element 1100 is fixed to the movable body 1200. The movable body 1200 moves on a plane perpendicular to the incident direction of light incident on the imaging element 1100, as described below. A lens unit 1110 is disposed above the imaging element 1100. The imaging element 1100 is connected to one end of a flexible printed circuit board 1111, and a connector 1112 is connected to the other end of the flexible printed circuit board 1111. Signals output from the imaging element 1100 are guided to the connector 1112 via the flexible printed circuit board 1111. Furthermore, control signals for the imaging element 1100 are input to the connector 1112 and supplied to the imaging element 1100 via the flexible printed circuit board 1111.

[0072] Although not shown here, the pixels included in the imaging element 1100 are provided with red (R), green (G), and blue (B) color filters, and are arranged in blocks according to a Bayer arrangement, with each block comprising four pixels: two horizontal pixels × two vertical pixels. In the Bayer arrangement, two pixels with a green color filter, one pixel with a red color filter, and one pixel with a blue color filter are arranged such that pixels with color filters of the same color are not adjacent to each other.

[0073] In the following text, unless otherwise stated, a pixel with a green filter is referred to as a G pixel, a pixel with a red filter as an R pixel, and a pixel with a blue filter as a B pixel. Furthermore, red is referred to as R color, green as G color, and blue as case color.

[0074] exist Figure 2Part (b) is a perspective view showing an example of the appearance when the lens unit 1110 is removed from the sensor module 1000. The base 1300 is provided with magnets 1301a and 1301b. Magnets 1301a and 1301b may be permanent magnets. In the example of part (b), magnets 1301a and 1301b are disposed in respective portions separated by the components of the base 1300 in the y-direction of the figure. Furthermore, a yoke 1302a is disposed at the bottom relative to magnet 1301a. Although not shown in the figure, the yoke is also disposed on the surface of magnet 1301b facing the partition.

[0075] Figure 3 This is a perspective view showing an example of the structure of the sensor module 1000 according to an embodiment in an exploded manner.

[0076] exist Figure 3 In this embodiment, the movable body 1200 has a panel 1201c made of magnetic material at a position corresponding to the magnet 1301a of the base 1300. Similarly, the movable body 1200 has panels 1201a and 1201b on a surface facing the origin along the y-axis. Panels 1201a to 1201c are, for example, steel plates. Panels 1201a to 1201c only need to be made of magnetic material, and the material is not limited to steel.

[0077] The movable body 1200 has a V-shaped groove 1202 on one side of the surface (second surface) facing the base 1300 along the y-axis. In addition, the movable body 1200 has a recessed portion 1203 at the substantially middle part on the other side along the y-axis.

[0078] The base 1300 has holes 1303a, 1303b, and 1303c in its surface facing the movable body 1200 (first surface). When the movable body 1200 is correctly positioned relative to the base 1300, holes 1303a and 1303b are positioned corresponding to the V-shaped groove 1202. When the movable body 1200 is correctly positioned relative to the base 1300, hole 1303c is positioned corresponding to the recess 1203. Perfectly spherical members 1210a, 1210b, and 1210c are respectively placed in holes 1303a, 1303b, and 1303c.

[0079] The movable body 1200 is disposed relative to the base 1300 via perfectly spherical members 1210a to 1210c. That is, perfectly spherical members 1210a and 1210b are placed in the holes 1303a and 1303b respectively, abutting against the V-shaped groove 1202, and are sandwiched between the movable body 1200 and the base 1300.

[0080] Figure 4This is a schematic diagram showing the state in which the perfectly spherical member 1210a, according to an embodiment, is sandwiched between the moving body 1200 and the base 1300. The perfectly spherical member 1210b is similar to the perfectly spherical member 1210a, and therefore its description is omitted here.

[0081] exist Figure 4 In this embodiment, the opening of the hole 1303a in the surface of the base 1300 facing the movable body 1200 has the shape of a conical surface 1304. In this example, the bottom of the hole 1303a is cylindrical, but this disclosure is not limited to this example, and the bottom of the hole 1303a may also have a conical surface 1304. When the perfectly spherical member 1210a is placed in the hole 1303, the spherical surface abuts against the conical surface 1304, and the position of the perfectly spherical member 1210a in the base 1300 is fixed. Furthermore, by forming the opening of the hole 1303a as a conical surface 1304, the perfectly spherical member 1210a can rotate freely in that position.

[0082] In the hole 1303a, with the perfectly spherical member 1210a placed within it, the member 1210a is sandwiched between the movable body 1200 and the base 1300, such that the member 1210a also abuts against the surface of the V-shaped groove 1202. Similarly, in the hole 1303b, with the perfectly spherical member 1210b placed within it, the member 1210b is sandwiched between the movable body 1200 and the base 1300, such that the member 1210b also abuts against the surface of the V-shaped groove 1202. Therefore, in this state, the movable body 1200 is movable relative to the base 1300 only along the direction of the V-shaped groove 1202.

[0083] On the other hand, the recess 1203 is formed such that when the perfectly spherical member 1210c placed in the hole 1303c abuts against the upper surface of the recess 1203, the height of the abutment position of the perfectly spherical member 1210c is equal to the height of the abutment positions of the perfectly spherical members 1210a and 1210b when they are placed in the holes 1303a and 1303b and abut against the V-shaped groove 1202. Therefore, the surfaces (first surface and second surface) facing each other of the base 1300 and the moving body 1200 are parallel to each other. The diameter of the recess 1203 is preferably determined such that when the moving body 1200 moves, the perfectly spherical member 1210c does not contact the edge of the recess 1203.

[0084] The perfectly spherical components 1210a to 1210c are preferably made of non-magnetic materials, particularly high-hardness ceramics such as alumina or zirconium oxide. Furthermore, by filling the space between the V-groove 1202 and the perfectly spherical components 1210a and 1210b with grease or the like, vibrations during operation can be suppressed, and the durability of the sliding portion (V-groove 1202) can be improved.

[0085] Figure 5 and Figure 6 This is a schematic diagram showing more specifically an example of the configuration of the mobile body 1200 and the base 1300 according to an embodiment. Figure 5 Part (a) shows the state of the moving body 1200 as viewed from above, i.e., from the side where the imaging element 1100 is fixed, and part (b) shows a cross-section taken along line A-A' in part (a). Note that in part (a), a portion is shown in a semi-transparent manner for ease of description.

[0086] exist Figure 5 In part (a), the piezoelectric element 1400 is supplied with a drive signal via lead 1401. The positions where the perfectly spherical members 1210a, 1210b, and 1210c are located (the positions of the holes 1303a, 1303b, and 1303c) are located at the center of the triangle formed by the positions of the perfectly spherical members 1210a, 1210b, and 1210c.

[0087] exist Figure 5 In part (b), magnets 1301a are disposed on both sides of piezoelectric element 1400, and yokes 1302a and 1302b are disposed on the respective sides. On the other hand, as described above, a panel 1201c, which is a magnetic material (e.g., a steel plate), is disposed on the movable body 1200 at a position corresponding to magnet 1301a. Therefore, the movable body 1200 is attracted toward the base 1300 by the magnetic force 1305 of magnet 1301a. Thus, the positional relationship of the movable body 1200 relative to the base 1300 in the height direction (z-axis direction) is maintained.

[0088] exist Figure 6 Part (b) shows the state of the moving body 1200 as viewed from above, and part (a) shows a cross section taken along line B-B' in part (b). Note that in part (b), the surface of the moving body 1200 to which the imaging element 1100 is fixed is omitted for ease of description.

[0089] In section (b), the piezoelectric element 1400 expands and contracts in the y-axis direction of the figure, and the end surface of the protrusion 1220 of the moving body 1200 abuts against the end surface of the piezoelectric element 1400 in the expansion and contraction direction. Therefore, as shown in section (b), the moving body 1200 moves in the y-axis direction of the figure according to the expansion and contraction of the piezoelectric element 1400. In addition, in the base 1300, magnets 1301b are provided on both sides of the protrusion 1220, and yokes 1302b are provided on each side. On the other hand, the moving body 1200 has panels 1201a and 1201b, which are magnetic materials (e.g., steel plates), provided at positions corresponding to the magnets 1301b. Therefore, the moving body 1200 is attracted toward the piezoelectric element 1400 by the magnetic force 1306 of each magnet 1301b. Therefore, the moving body 1200 moves in the y-axis direction in the figure, following the expansion and contraction of the piezoelectric element 1400.

[0090] Figure 7 This is a schematic diagram illustrating the movement of the movable body 1200 according to an embodiment. The magnetic forces 1305 and 1306 of magnets 1301a and 1301b inhibit the movement of the movable body 1200 in directions other than the direction of travel (the y-axis direction in the figure) corresponding to the expansion and contraction of the piezoelectric element 1400. That is, in the embodiment, the effect that is usually obtained by using a spring is obtained by magnetic force.

[0091] Therefore, movement of the moving body 1200 in directions other than the direction of travel can be suppressed in a non-contact manner, which improves the durability of the contact portion and also eliminates frictional noise caused by the movement of the moving body 1200, thereby achieving noise reduction. In addition, since the springs used in related technologies are eliminated, the size can be reduced compared to the configurations of related technologies.

[0092] Furthermore, by using components made of non-magnetic materials as fully spherical components 1210a to 1210c placed between the movable body 1200 and the base 1300, the fully spherical components will not be attracted by magnets 1301a and 1301b when assembling the sensor module 1000, which improves the ease of assembly.

[0093] 2-2. Configuration and control of the imaging apparatus according to embodiments of the present disclosure

[0094] Next, the configuration and control of the imaging apparatus according to embodiments of this disclosure will be described.

[0095] 2-2-1. Configuration according to the implementation method

[0096] First, the configuration of the imaging apparatus according to the embodiment will be described.

[0097] Figure 8 This is an exemplary block diagram illustrating a general configuration of an imaging apparatus for performing video perspective according to the prior art. Figure 8 In this context, an electronic device 500 employing an imaging apparatus according to the prior art includes an imaging unit 510, an imaging processing unit 520, a display control unit 530, a display 540, and a control unit 550.

[0098] The control unit 550 includes a processor, such as a central processing unit (CPU), and controls the overall operation of the electronic device 500 according to a program. The control unit 550 may include a signal generation unit that generates synchronization signals for operating the image sensor 512 and the imaging processing unit 520.

[0099] Imaging unit 510 includes lens unit 511 and image sensor 512. Image sensor 512 includes a pixel array as imaging elements, wherein pixels that output signals according to incident light are arranged in a two-dimensional dot matrix pattern. Lens unit 511 collects light from the subject and uses this light to illuminate the light-receiving surface of image sensor 512. Image sensor 512 converts analog signals (pixel signals) corresponding to the light received on the light-receiving surface into digital image data and outputs the digital image data to imaging processing unit 520.

[0100] The imaging processing unit 520 includes a signal processing unit 521 and an image processing unit 522. The signal processing unit 521 performs predetermined signal processing, such as demosaicing, on the pixel data output from the imaging unit 510 to generate image data for each frame. The image processing unit 522 performs predetermined image processing, such as tone correction, on the image data generated by the signal processing unit 521. The image processing unit 522 outputs the image data after image processing to the display control unit 530.

[0101] The signal processing unit 521 and the image processing unit 522 can be composed of an image signal processor (ISP) and a digital signal processor (DSP), respectively.

[0102] The display control unit 530 generates a display signal that can be displayed by the display 540 based on the image data output from the image processing unit 522. The display 540 displays a screen corresponding to the display signal generated by the display control unit 530 based on the image data.

[0103] When the resolution of the image sensor 512 increases in such a configuration, the amount of image data transmitted from the image sensor 512 to the imaging processing unit 520 per frame increases, and there is a possibility that data transmission cannot be completed within a single frame period. In such a case, in order to complete the data transmission of a single frame within a single frame period, methods such as reducing the frame rate or sparsifying the image data of a single frame are required. However, these methods reduce the quality of the displayed image, making it impossible to utilize high resolution.

[0104] Figure 9 This is an exemplary block diagram illustrating the configuration of an imaging apparatus that performs video perspective using super-resolution, applicable to embodiments of this disclosure. Figure 9 In this context, the electronic device 1 employing an imaging apparatus according to the prior art includes an imaging unit 100, an imaging processing unit 120, a display control unit 130, a display 140, and a control unit 150. The imaging apparatus according to the embodiment may include... Figure 9 The configuration includes an imaging unit 100 and an imaging processing unit 120.

[0105] The control unit 150 includes a processor such as a CPU and memories such as read-only memory (ROM) and random access memory (RAM), and controls the overall operation of the electronic device 500 according to a program. The control unit 150 may also include a clock signal generation unit (not shown) that generates clock signals to operate the image sensor 110 and the imaging processing unit 120. The clock signals generated by the clock signal generation unit are supplied to the imaging unit 100, the imaging processing unit 120, and the display control unit 130.

[0106] Imaging unit 100 and reference Figures 2 to 7 Corresponding to the described sensor module 1000, it includes an image sensor 110, a lens unit 111, a drive unit (DRV) 112, and an actuator (ACT) 113.

[0107] Image sensor 110 and reference Figures 2 to 7 Corresponding to the described imaging element 1100, the image sensor 110 includes a pixel array unit as an imaging element, wherein pixels that output signals according to incident light are arranged in a two-dimensional dot matrix pattern, and the image sensor 110 is fixedly disposed at the moving body 1200. The image sensor 110 generates a vertical synchronization signal Vsync and a horizontal synchronization signal Hsync based on a clock signal supplied from a clock signal generation unit, drives the pixel array unit based on the vertical synchronization signal Vsync and the horizontal synchronization signal Hsync, and captures an image (exposure).

[0108] The image sensor 110 converts the analog signal (pixel signal) read from each pixel based on the light received on the light-receiving surface into digital pixel data and outputs the digital pixel data. The analog signal (pixel signal) is captured from each pixel of the pixel array. The pixel data output from the image sensor 110 is supplied to the imaging processing unit 120.

[0109] The drive unit 112 generates a drive signal for driving the actuator 113 based on, for example, a vertical synchronization signal Vsync generated by the image sensor 110. The actuator 113 includes a reference... Figures 2 to 7 The described piezoelectric element 1400 is driven by a drive signal generated by the drive unit 112, and moves the position of the image sensor 110. Each time the position is moved based on the drive signal, the image sensor 110 captures an image.

[0110] The imaging processing unit 120 includes a signal processing unit 121 and an image processing unit 122, which are configured in a manner similar to... Figure 8 Examples of these configurations are provided by, for example, an ISP and a DSP. The signal processing unit 121 performs predetermined signal processing, such as demosaicing, on the pixel data output from the imaging unit 510 to generate image data for each frame. The image processing unit 122 performs predetermined image processing, such as tone correction, on the image data generated by the signal processing unit 121. The image processing unit 122 outputs the image data after image processing to the display control unit 130.

[0111] Figure 10 This is a schematic diagram illustrating the imaging control performed by the imaging apparatus according to an embodiment. Figure 10 In this image sensor 110, images are captured according to a frame period based on a vertical synchronization signal Vsync and a horizontal synchronization signal Hsync (not shown). For each frame, the image sensor 110 outputs the pixel data obtained through capture. That is, the image sensor 110 alternately outputs pixel data for each odd-numbered frame (odd number) and pixel data for each even-numbered frame (even number).

[0112] On the other hand, the drive unit 112 generates a drive signal to drive the actuator 113 based on the vertical synchronization signal Vsync generated by the image sensor 110, and shifts the position of the image sensor 110 for each frame. Here, it is assumed that the position of the image sensor 110 is as shown in the reference... Figure 1A The described shift amount is 1 / 2 pixel. Figure 10In the example, in each even-numbered frame (even), the image sensor 110 captures an image at a position shifted by 1 / 2 pixel in each direction, both horizontally and vertically, from position Posi#1 in each odd-numbered frame (odd).

[0113] The imaging processing unit 120 alternately outputs image data (image #A) of odd-numbered frames (odd-numbered frames) and image data (image #B) of even-numbered frames (even-numbered frames) according to the frame period.

[0114] Return to reference Figure 9 The display control unit 130 generates display signals to display on the display 140 the image data (image #A) of the odd-numbered frame (odd number) and the image data (image #B) of the even-numbered frame (even number) output from the imaging processing unit 120.

[0115] Figure 11 This is a schematic diagram illustrating the display signal according to the embodiment. As shown, the display control unit 130 alternately outputs image data (image #A) of odd-numbered frames (odd-numbered frames) and image data (image #B) of even-numbered frames (even-numbered frames) to the display 140 according to the frame period. Image #A based on the image data of odd-numbered frames (odd-numbered frames) and image #B based on the image data of even-numbered frames (even-numbered frames) are shifted by 1 / 2 pixel in each of the horizontal and vertical directions.

[0116] Therefore, when images #A and #B displayed on monitor 140 are observed, due to the persistence of vision, image #A and image #B, which are shifted by 1 / 2 pixel in each of the horizontal and vertical directions relative to image #A, are observed as if these images were displayed simultaneously, as shown in reference. Figure 1B As described in part (b). Therefore, a super-resolution image can be observed, which has a resolution twice as high as that captured by the image sensor 110. The amount of image data transmitted from the image sensor 110 to the imaging processing unit 120 is the amount of image data with a resolution of 1 / 2 that observed in each of the horizontal and vertical directions.

[0117] Figure 12 This is an exemplary block diagram illustrating the hardware configuration of an electronic device 1 that employs an imaging apparatus according to an embodiment.

[0118] exist Figure 12In this device, electronic device 1 includes a CPU 600, a ROM 601, a RAM 602, a storage device 603, a data I / F 604, and an input device 605. Electronic device 1 also includes a lens unit 610, a sensor module 1000, an ISP 612, a DSP 613, and a display control unit 614. These units included in electronic device 1 are communicatively connected to each other via a bus 620.

[0119] The display 630 is connected to the display control unit 614. The display 630 may be included in the electronic device 1, or it may be externally connected to the electronic device 1.

[0120] Storage device 603 is a non-volatile storage medium such as flash memory or hard disk drive. CPU 600 uses RAM 602 as working memory to control the overall operation of electronic device 1 according to the program stored in ROM 601 and storage device 603.

[0121] The Data I / F 604 is an interface for sending and receiving data to and from external devices. The Data I / F 604 can send and receive data via wired or wireless connections. The Data I / F 604 can communicate via communication networks such as the Internet or a Local Area Network (LAN).

[0122] Input device 605 is a device for receiving user operations, such as various switches, buttons and touch panels.

[0123] Lens unit 610 includes a condenser lens and may also include an autofocus mechanism, an auto aperture mechanism, and a zoom mechanism. Lens unit 610 collects light from the subject and uses this light to illuminate sensor module 1000. Sensor module 1000 outputs image data corresponding to the illuminated light to ISP 612. For example, ISP 612 and DSP 613 perform processing corresponding to signal processing unit 121 and image processing unit 122 described above, respectively. Display control unit 614 corresponds to display control unit 130 described above and generates a display signal that can be displayed by display 630 based on the image data output from DSP 613. Display 630 displays an image corresponding to the display signal generated by display control unit 614.

[0124] 2-2-2. Control according to the implementation method

[0125] The control according to the implementation method will be described in more detail below.

[0126] Figure 13 This is a schematic diagram illustrating the effect of super-resolution that can be applied to the implementation. Here, the image sensor 110 is assumed to capture images at 2K resolution.

[0127] exist Figure 13 As shown in section (a), image #A is captured in each odd-numbered frame (odd). In each even-numbered frame (even), the position of image sensor 110 is shifted by 1 / 2 pixel in each direction (horizontal and vertical) relative to its position when captured in the odd-numbered frame (odd), and image #B is captured. In this way, images #A and #B of the two odd-numbered and even-numbered frames captured at 2K resolution are displayed sequentially in a time series. Image #B is an image shifted by 1 / 2 pixel in each direction (horizontal and vertical) relative to image #A.

[0128] When a person observes images #A and #B displayed alternately according to a frame period, images #A and #B are composited due to the persistence of vision and recognized as a high-resolution image exceeding 2K resolution, such as... Figure 13 As shown in part (b).

[0129] Figure 14 This is an exemplary timing diagram used to describe the processing of super-resolution capture and display that can be applied to implementation methods.

[0130] exist Figure 14 In this diagram, the drive unit 112 generates a drive signal 70 for driving the actuator 113 in synchronization with a vertical synchronization signal Vsync (not shown), and supplies the drive signal 70 to the actuator 113. For example, when the drive signal 70 is high, the actuator 113 moves the image sensor 110 to position Posi#1, and when the drive signal 70 is low, the actuator 113 moves the image sensor 110 to position Posi#2, which is shifted by 1 / 2 pixel in each of the horizontal and vertical directions relative to position Posi#1. In the example in this figure, position Posi#2 is shifted to the lower right relative to position Posi#1 in the figure.

[0131] Image sensor 110 repeatedly captures image #A (or image #A') in each odd-numbered frame (odd-numbered frame) and image #B (or image #B') in each even-numbered frame (even-numbered frame) at frame timing synchronized with the vertical synchronization signal Vsync. That is, in Figure 14 In the example, image sensor 110 captures an image in even-numbered frame (even-numbered) at a position shifted 1 / 2 pixel to the lower right of the odd-numbered frame (odd-numbered) in the figure.

[0132] exist Figure 14 In this context, we assume that images #A and #B are consistent with the description above. Figure 13The corresponding reference images are shown, and images #A' and #B' are the actual captured images. In the following text, to avoid complexity, unless otherwise stated, image #A or #A' will be represented by image #A', and image #B or #B' will be represented by image #B'.

[0133] Image sensor 110 transmits the acquired image data (raw (RAW) data) to imaging processing unit 120. Since the transmission rate is limited in the communication between image sensor 110 and imaging processing unit 120, image sensor 110 transmits the image data of each of images #A' and #B' to imaging processing unit 120 in a time-division manner. Here, time-division can mean that images #A' and #B' are transmitted sequentially.

[0134] The imaging processing unit 120 performs predetermined processing, such as demosaicing or super-resolution processing, on the image data of each of the images #A' and #B' transmitted from the image sensor 110. Here, in super-resolution processing, for example, since image #B' is offset by half a pixel relative to image #A', the position of each pixel in image #B' is shifted by half a pixel to correct the offset.

[0135] The imaging processing unit 120 displays images #A' and #B', whose pixel positions are shifted in this manner, sequentially on the display 140 in a time-division manner. Therefore, due to the persistence of vision, an image with a higher resolution than that of the image sensor 110 can be observed.

[0136] Figure 15 It is a schematic diagram used to describe the configuration and processing flow of the imaging apparatus according to the embodiments.

[0137] exist Figure 15 In this process, the drive unit 112 generates a drive signal 70 in the form of a rectangular wave to drive the actuator 113, based on timing control of a synchronization signal 80, such as a vertical synchronization signal Vsync, output from the sensor module 1000. The actuator 113 is driven to move the moving body 1200 in a predetermined direction according to the expansion and contraction of the piezoelectric element 1400 controlled by the drive signal 70 supplied from the drive unit 112.

[0138] Note that the sensor module 1000 may be equipped with a position sensor 200 that detects the position of the moving body 1200 or the imaging element 1100 fixed to the moving body 1200. The drive unit 112 can correct the amount of movement of the moving body 1200 caused by the actuator 113 by controlling the amplitude of the drive signal 70 based on the detection result from the position sensor 200.

[0139] In sensor module 1000, image sensor 110 captures an image whenever moving body 1200 moves according to drive signal 70. For example, sensor module 1000 captures an image according to synchronization signal 80 using light incident on imaging element 1100 via lens unit 1110. Sensor module 1000 outputs the image data (raw (RAW) data) obtained through capture to imaging processing unit 120. Imaging processing unit 120 performs image processing such as demosaicing on the image data output from sensor module 1000 and outputs the image data. Display control unit 130 displays an image based on the image-processed image data output from imaging processing unit 120 on display 140.

[0140] Figure 16 This is a schematic diagram illustrating the stability of positional movement of the image sensor 110 (imaging element 1100) according to an embodiment. Figure 16 In the diagram, the horizontal axis represents time, and the vertical axis represents position relative to a predetermined zero point. Feature line 50 indicates the position of image sensor 110.

[0141] exist Figure 16 In the middle, assuming at time... , , , , , …the drive signal 70 is switched between high and low states, causing the moving body 1200 to move. As shown in the figure, after the moving body 1200 (image sensor 110) moves, the position of the image sensor 110 immediately becomes unstable. Therefore, the image sensor 110 is controlled to capture images during each period 51 in which the position of the image sensor 110 is stable.

[0142] Figure 17 This is a schematic diagram illustrating an example of controlling the capture timing of an image sensor 110 according to an embodiment. Figure 17 In the diagram, the horizontal axis represents time. The image sensor 110 begins exposure (capture) synchronously with a synchronization signal 80, such as a vertical synchronization signal Vsync, and ends exposure at a predetermined time within a frame period. The image sensor 110 reads out the signal of each pixel whose exposure has ended during the period from the end of the exposure to the start of the exposure of the next frame.

[0143] The drive unit 112 delays the rise time of the vertical synchronization signal Vsync by a predetermined delay time. The timing of the rise and fall of the drive signal 70 is determined by the timing of the drive unit 112. That is, the drive unit 112 generates the drive signal 70, causing the moving body 1200 to move during each period when the image sensor 110 is not performing an exposure, and causing the exposure to begin and end within each period 51, during which the position of the image sensor 110 is stable. More specifically, the drive unit 112 generates the drive signal 70 with a rise time delayed from the vertical synchronization signal Vsync. The timing of the rise and fall of the drive signal 70 is recorded. The actuator 113 moves the moving body 1200 according to the rise and fall of the drive signal 70.

[0144] 2-2-3. Example of pixel shift capture according to the implementation method

[0145] Next, the pixel shift capture according to the embodiment will be described in more detail. In the pixel shift capture according to the prior art, as shown in example... Figure 1A and Figure 1B As described, the imaging element moves by the same amount in both the horizontal and vertical directions. In contrast, in embodiments of this disclosure, pixel shift capture is performed such that the amount of movement in the horizontal direction is different from the amount of movement in the vertical direction.

[0146] Figure 18 This is a schematic diagram used to describe pixel shift capture according to an embodiment. Figure 18 In sections (a) to (c), each right figure illustrates the pixel shifting pattern of pixel 10 included in pixel array 11, and each left figure illustrates the drive signal 70. Furthermore, it is assumed that pixel 10 is arranged in a Bayer arrangement based on the color of the color filter, and that in the 2×2 pixel arrangement, the top-left pixel is an R pixel, the bottom-right pixel is a B pixel, and the top-right and bottom-left pixels are G pixels.

[0147] Figure 18 Part (a) illustrates a first mode example of pixel shift capture. In this first mode example, a two-step pixel shift, including a state where no pixel shift is performed, is performed by shifting the pixel array 11 (imaging element 1100) by one pixel in the horizontal direction and by two pixels in the vertical direction. The drive signal 70a is a simple rectangular wave with a signal level for not performing the shift and a signal level for tilting the pixel array 11 by one pixel in the horizontal direction and by two pixels in the vertical direction.

[0148] Figure 18Part (b) illustrates a second mode example of pixel shift capture. In this second mode example, a four-step pixel shift is performed by repeating the pattern of shifting pixel array 11 by 1 / 2 pixel horizontally and by 1 pixel vertically three times. The drive signal 70b is a signal with a signal level that changes progressively in three steps, and the signal level in each of the three steps is the level used to tilt pixel array 11 by 1 / 2 pixel horizontally and by 1 pixel vertically.

[0149] Figure 18 Section (c) illustrates a third mode example of pixel shift capture. In this third mode example, an eight-step pixel shift is performed by repeating the pattern of shifting pixel array 11 by 1 / 4 pixel horizontally and 1 / 2 pixel vertically seven times. The drive signal 70c is a signal with a signal level that changes progressively in seven steps, and the signal level in each of the seven steps is the level used to tilt pixel array 11 by 1 / 4 pixel horizontally and 1 / 2 pixel vertically.

[0150] In any of the first to third modes, the amount of pixel shifting differs between a first direction (e.g., row direction) along the two-dimensional matrix of pixels 10 arranged in the pixel array 11 and a second direction (e.g., column direction) different from the first direction, and the pixel array 11 is moved obliquely relative to the two-dimensional matrix. Additionally, in any of the first to third modes, pixel shifting is performed in a direction where the G pixels do not overlap after the shift. Furthermore, in any of the first to third modes, the pixel array 11 is moved to each of the division positions obtained by dividing a period, in which the original arrangement is reproduced by shifting the pixel array 11 within that period.

[0151] Referring to the first mode example shown in section (a), for example, when pixel array 11 is shifted by one pixel in the horizontal direction and two pixels in the vertical direction as shown in the first mode example, the relative positional relationship between pixels R, G, and B remains unchanged before and after the shift. However, for example, the shifted pixel G is located at the position of the shifted pixel R, and the absolute positional relationship of the shifted pixel arrangement relative to the previous pixel arrangement changes, and the original arrangement is not reproduced. On the other hand, when pixel array 11 is shifted by two pixels in the horizontal direction and four pixels in the vertical direction, the shifted pixel R is located at the position of the previous pixel R, and for example, the absolute positional relationship of the shifted pixel arrangement relative to the previous pixel arrangement remains unchanged, and the original arrangement is reproduced.

[0152] In this way, when the pixel array 11 is tilted relative to the two-dimensional dot matrix, for example, in the case of a two-pixel × two-pixel Bayer arrangement, the absolute positional relationship relative to the original arrangement remains unchanged, and the period of reproducing this arrangement is a period of two pixels in the horizontal direction and four pixels in the vertical direction. Therefore, in the first mode example of part (a), the shifted position of the pixel array 11 is the position obtained by dividing the period into two. In the second mode example of part (b), the shifted position of the pixel array 11 is the respective division position obtained by dividing the period into four. In the third mode example of part (c), the shifted position of the pixel array 11 is the respective division position obtained by dividing the period into eight.

[0153] Next, the direction in which the pixel array 11 will be shifted to effectively achieve super-resolution will be described. Figure 19A and Figure 19B In the diagram, each left image shows the pixel arrangement and pixel shift direction, and each right image shows the spatial resolution. Additionally, in... Figure 19A and Figure 19B In the image, R, G, and B pixels are arranged in a Bayer arrangement, where in a 2-pixel × 2-pixel grid, the top left and bottom right pixels are G pixels, the top right pixel is an R pixel, and the bottom left pixel is a B pixel.

[0154] Figure 19A This is a schematic diagram illustrating the spatial resolution in a Bayer arrangement when pixel shifting is not performed. In this case, as... Figure 19A As shown on the upper right, the spatial resolution of each of the R and B colors is one unit in each of the vertical, horizontal, and diagonal directions, and the spatial resolution of the G color is two units in each of the vertical and horizontal directions and one unit in each diagonal direction. Note that the dashed square outline represents an example of a full setup of four R pixels, four B pixels, and four G pixels, and indicates the ideal spatial resolution in a 2-pixel × 2-pixel Bayer arrangement.

[0155] Figure 19B This is a schematic diagram illustrating the spatial resolution in the Bayer arrangement when pixel shifting is performed. Figure 19B Part (a) shows an example of performing a pixel shift of one pixel in the horizontal direction. In this case, the spatial resolution of each of the R and B colors is one unit in the vertical direction and two units in the horizontal direction, resulting in an asymmetry between the vertical and horizontal directions. On the other hand, the spatial resolution of the G color is two units in each of the vertical and horizontal directions, resulting in an ideal spatial resolution.

[0156] Figure 19BPart (c) shows the angle at 45 degrees. In the direction of the arrangement of G pixels, in other words, an example of performing a pixel shift of one pixel. In this case, the spatial resolution of each of the R and B colors is two units in each of the vertical and horizontal directions, and one unit in each diagonal direction. The spatial resolution of the G color is... Figure 19A The spatial resolution is the same as in the example where pixel shifting is not performed.

[0157] Figure 19B Part (d) shows an example of performing a pixel shift of one pixel in each of the four directions. In this case, the spatial resolution of each of the R, B, and G colors is two units in each of the vertical and horizontal directions, thus producing the desired spatial resolution.

[0158] In these examples, Figure 19B The example shown in section (c) that performs pixel shifting in four directions achieves the highest spatial resolution, but the mechanism used to implement the pixel shifting is complex. On the other hand, in the examples of performing pixel shifting in one direction in sections (a) and (b), the example in section (a) results in asymmetry between the vertical and horizontal directions, while the example in section (b) achieves balance in each direction. Based on the above, performing pixel shifting in the tilt direction is considered preferred.

[0159] 2-2-4. Specific example of pixel shift capture according to the implementation method

[0160] Next, the pixel shift capture according to the embodiments of this disclosure will be described in more detail.

[0161] Figures 20A to 20C This is a schematic diagram used to describe pixel shift capture according to an embodiment. Figure 20A An example of a Bayer arrangement without pixel shifting is shown. Figure 20A It is similar to the description above. Figure 19A The graph is presented for comparison.

[0162] Figure 20B It shows in Figure 18 The first mode example of pixel shift capture described in section (a), and as shown in section (b), the pixel array 11 (imaging element 1100) is shifted by one pixel in the horizontal direction and by two pixels in the vertical direction. Furthermore, as shown in section (a), the drive signal 70a is as follows: Figure 18 In part (a) of the example, there is a simple rectangular wave with a signal level that changes in two steps.

[0163] Figure 20B Part (c) shows an example where pixel 10, in the case of pixel shift capture performed in the first mode example shown in Part (b), is superimposed as a 4-pixel × 4-pixel block, with focus on each of the R, B, and G pixels. The left image of Part (c) shows an example with focus on R pixels, and all pixels in the first and third rows have the R color. The middle image of Part (c) shows an example with focus on B pixels, and all pixels in the second and fourth rows are B pixels. Additionally, the right image of Part (c) shows an example with focus on G pixels, and all pixels in the 4-pixel × 4-pixel block are G pixels.

[0164] Figure 20B Part (d) illustrates the spatial resolution in this first mode example. In this case, the spatial resolution of each of the R and B colors is one unit in the vertical direction and two units in the horizontal direction, resulting in an asymmetry between the vertical and horizontal directions. On the other hand, the spatial resolution of the G color is two units in each of the vertical and horizontal directions, thus producing an ideal spatial resolution.

[0165] Figure 20C An example of pixel shift capture is shown, in which a pixel is shifted by one pixel in the horizontal direction, as shown in section (b). As shown in section (a), the drive signal 70a' is a simple rectangular wave with a signal level for causing the pixel array 11 to shift by one pixel in the horizontal direction.

[0166] Figure 20C Part (c) illustrates the following example: In the case of performing pixel shift capture for a pixel shifted one pixel horizontally as shown in Part (b), pixel 10 is superimposed into a 4-pixel × 4-pixel block, where each of the R, B, and G pixels is focused. In this case, the same pattern example as the first pattern example described above is obtained. That is, in the example focused on R pixels on the left of Part (c), all pixels in the first and third rows have the R color, and in the example focused on B pixels in the middle of Part (c), all pixels in the second and fourth rows are B pixels. In the example focused on G pixels on the right of Part (c), all pixels in the 4-pixel × 4-pixel block are G pixels.

[0167] Figure 20CPart (d) illustrates the spatial resolution when pixel shift capture is performed for a pixel shifted one pixel horizontally as shown in Part (b). In this case, the spatial resolution of each of the R and B colors is one unit vertically and two units horizontally, and the spatial resolution of the G color is two units in each of the vertical and horizontal directions, thus producing the ideal spatial resolution. This spatial resolution is equivalent to... Figure 20B Spatial resolution in part (d).

[0168] As from Figure 20B Part (d) and Figure 20C A comparison between parts (d) shows that there is no difference in spatial resolution between performing pixel shift capture on a pixel shifted one pixel horizontally and two pixels vertically and performing pixel shift capture on a pixel shifted only one pixel horizontally. However, according to Figure 20B The pixel shift capture shown in the first mode example is scalable, in which a pixel is shifted by one pixel in the horizontal direction and by two pixels in the vertical direction.

[0169] Figure 21 This is a schematic diagram illustrating an extended example of pixel shift capture according to a first mode example of an implementation. Figure 21 Examples and Figure 18 Corresponding to the second mode example of pixel shift capture described in part (b), the mode that causes pixel array 11 to shift by 1 / 2 pixel in the horizontal direction and by 1 pixel in the vertical direction is repeated three times. The drive signal 70b is a signal with a progressively changing signal level, and the signal level of each of the three steps is the level used to tilt pixel array 11 by 1 / 2 pixel in the horizontal direction and by 1 pixel in the vertical direction.

[0170] Figure 21 Part (c) illustrates an example where pixels 10 are stacked into a 4-pixel × 4-pixel block, with focus on each of the R, B, and G pixels, in a pattern repeated three times as shown in Part (b), where pixel array 11 is shifted by 1 / 2 pixel horizontally and by 1 pixel vertically. The left image of Part (c) shows an example with focus on R pixels, and all pixels in the 4-pixel × 4-pixel block are of color R. The middle image of Part (c) shows an example with focus on B pixels, and all pixels in the 4-pixel × 4-pixel block are of color B. The right image of Part (c) shows an example with focus on G pixels, and all pixels in the 4-pixel × 4-pixel block are G pixels.

[0171] Figure 21 Part (d) illustrates the spatial resolution in this second mode example. In this case, the spatial resolution of each of the R, B, and G colors is two units in the vertical direction and four units in the horizontal direction, and the spatial resolution in each of the horizontal and vertical directions is higher than that according to Figure 20B The spatial resolution of the first mode example shown in part (b).

[0172] Figure 22 This is a schematic diagram illustrating an example of increasing the number of steps in the change of the drive signal 70 according to an embodiment.

[0173] exist Figure 22 Part (a) shows an example where pixel shifting is not performed on pixels 10 arranged in a Bayer arrangement. Although not shown, in this example, since pixel shifting is not performed, the drive signal 70 does not change. This state is defined as a one-step change. See reference... Figure 19A As described, the spatial resolution of each of the R and B colors is one unit in each of the vertical, horizontal, and diagonal directions, and the spatial resolution of the G color is two units in each of the vertical and horizontal directions and one unit in each diagonal direction.

[0174] Figure 22 Part (b) shows in Figure 18 The first mode example of pixel shift capture described in part (a) includes a two-step pixel shift performed by a drive signal 70a in the form of a simple rectangular wave, the drive signal 70a having a signal level for not performing a shift and a signal level for tilting the pixel array 11 by one pixel in the horizontal direction and by two pixels in the vertical direction. (See also...) Figure 20B As described, the spatial resolution of each of the R and B colors is one unit in the vertical direction and two units in the horizontal direction, with the horizontal spatial resolution being twice as high as that in the example of part (a). On the other hand, the spatial resolution of the G color is two units in each of the vertical and horizontal directions, thus producing an ideal spatial resolution.

[0175] Figure 22 Part (c) shows in Figure 18The second mode example of pixel shift capture described in section (b) and the four-step pixel shift performed by a drive signal 70b having a signal level that changes in three steps, each of the three steps being a signal level used to tilt the pixel array 11 by 1 / 2 pixel in the horizontal direction and by 1 pixel in the vertical direction. See also... Figure 21 As described, the spatial resolution of each of the R, B, and G colors is two units in the vertical direction and four units in the horizontal direction, and compared with the example in part (b), the spatial resolution of each of the R and B colors is twice as high in each of the vertical and horizontal directions, and the spatial resolution of the G color is twice as high in the horizontal direction.

[0176] Figure 22 Part (d) shows in Figure 18 The third mode example of pixel shift capture described in part (c) involves performing eight-step pixel shifts via a drive signal 70c that changes signal level in seven steps, each of which is a signal level used to tilt the pixel array 11 by 1 / 4 pixel in the horizontal direction and by 1 / 2 pixel in the vertical direction. Although not shown, the spatial resolution of each of the R, G, and B colors is twice as high as the spatial resolution of part (c) in each of the vertical and horizontal directions.

[0177] Figure 22 Part (e) is a pattern example (referred to as the fourth pattern example) where the number of pixel shift steps is twice that of the example in part (d), and a 16-step pixel shift is performed by a drive signal 70d having a signal level that changes progressively in 15 steps, each of which is a signal level used to tilt the pixel array 11 by 1 / 8 pixel in the horizontal direction and by 1 / 4 pixel in the vertical direction. Although not shown, the spatial resolution of each of the R, G, and B colors is twice that of part (d) in each of the vertical and horizontal directions.

[0178] As from Figure 22 As can be seen from parts (a) to (e), even in pixel shifting along only one axis, super-resolution can be achieved in both the vertical and horizontal directions by increasing the number of pixel shift steps.

[0179] Figure 23 This is a schematic diagram illustrating the signal level of the drive signal 70 according to the number of pixel shift steps in an embodiment. Figure 23 In the middle, parts (a) to parts (e) are respectively with Figure 22Parts (a) to (e) correspond to each other. Description of pixel shifts at the lower part of parts (a) to (e) is omitted.

[0180] exist Figure 23 In each of sections (a) to (e), the above figure shows the signal level of the drive signal 70. As shown in section (a), the signal level has four levels from 0 to level Lv4. The signal level for linearly moving the pixel array 11 (imaging element 1100) a distance a is set to level Lv2—by distance a, the pixel array 11 is shifted by one pixel in the horizontal direction and by two pixels in the vertical direction, and the signal level for linearly moving the pixel array 11 a distance b is set to level Lv4—by distance b, the pixel array 11 is shifted by two pixels in the horizontal direction and by four pixels in the vertical direction.

[0181] Figure 23 Part (b) illustrates a two-step pixel shift of one pixel in the horizontal direction and two pixels in the vertical direction, according to a first pattern example of pixel shifting. In this case, the drive signal 70a is a simple rectangular wave with signal levels including: level 0, at which no shift is performed; and level Lv2, at which the pixel array 11 tilts and moves by one pixel in the horizontal direction and tilts and moves by two pixels in the vertical direction.

[0182] Figure 23 Part (c) illustrates a four-step pixel shift according to a second pattern example of pixel shifting, wherein the pattern of shifting pixel array 11 by 1 / 2 pixels horizontally and by 1 pixel vertically is repeated three times. In this case, the drive signal 70b is a signal with a signal level that changes progressively in three steps, and the signal level in each of the three steps is Lv1, which is half the level Lv2.

[0183] Figure 23 Part (d) illustrates an eight-step pixel shift according to a third pattern example of pixel shifting, wherein the pattern of shifting pixel array 11 by 1 / 4 pixel horizontally and 1 / 2 pixel vertically is repeated seven times. In this case, the drive signal 70c is a signal with a signal level that changes progressively in seven steps, and the signal level in each of the seven steps is half of level Lv1.

[0184] Figure 23 Part (e) shows an example of the fourth mode based on pixel shifting (see Figure 22The 16-step pixel shift in part (e) involves repeating a pattern 15 times in which the pixel array 11 is shifted 1 / 8 of a pixel horizontally and 1 / 4 of a pixel vertically. In this case, the drive signal 70d is a signal with a signal level that changes progressively in 15 steps, and the signal level in each of the 15 steps is 1 / 4 of level Lv1.

[0185] Although Figure 23 The signal level of each of the drive signals 70b to 70d shown in parts (c) to (e) is changed in a sawtooth waveform, but the change in signal level is not limited to this example, and the signal level may be changed, for example, in a triangular waveform. Figure 24A and Figure 24B These are schematic diagrams illustrating examples of how the signal level of the drive signal 70, according to an embodiment, is changed in a triangular waveform.

[0186] Figure 24A An example of changing the signal level of drive signal 70 in a second mode example of pixel shifting is shown, which refers to... Figure 23 The part (c) is described. Figure 24A Part (a) shows, as Figure 23 Example of a drive signal 70b in part (c) that has a signal level that changes in three steps with a sawtooth waveform.

[0187] Figure 24A Part (b) illustrates an example of a drive signal 70b' with a signal level that changes in three steps using a triangular waveform. Specifically, the drive signal 70b' changes its signal level until it reaches the position with the largest amount of movement among the division positions obtained by dividing the period of the original arrangement (in this example, the period of two horizontal pixels and four vertical pixels) obtained by pixel shifting of the pixel array 11 into four, while sparsifying the division positions one after another. Furthermore, the drive signal 70b' changes its signal level from the position with the largest amount of movement among the division positions toward the position with the smaller amount of movement in a manner corresponding to each sparsified position among the division positions.

[0188] Figure 24B An example of changing the signal level of drive signal 70 in a third mode example of pixel shifting is shown, which refers to... Figure 23 The part (d) is described. Figure 24B Part (a) shows, as Figure 23 Example of a drive signal 70c in part (d) that changes the signal level in 15 steps with a sawtooth waveform.

[0189] Figure 24APart (b) shows an example of a drive signal 70c' with a signal level that changes in 15 steps in a triangular waveform. Similar to the drive signal 70b' described above, specifically, the drive signal 70c' changes its signal level until it reaches the position with the largest amount of movement among the divided positions obtained by dividing the period of the original arrangement reproduced by pixel shifting of the pixel array 11 into 16 segments, while simultaneously sparsifying the divided positions one by one. Furthermore, the drive signal 70c' changes its signal level from the position with the largest amount of movement among the divided positions toward positions with smaller amounts of movement, corresponding to each sparsified position among the divided positions.

[0190] Compared to when the actuator 113 (piezoelectric element 1400) is driven by drive signals 70b and 70c with signal levels that change with a sawtooth waveform, the configuration of the actuator 113 (piezoelectric element 1400) is simplified when the actuator 113 (piezoelectric element 1400) is driven by drive signals 70b' and 70c' with signal levels that change with a triangular waveform with fewer harmonic components.

[0191] Note that Patent Document 1 discloses a structure comprising a shaft and a hole in a drive sliding part and clamped by a helical spring as a device for performing pixel shifting. However, in the configuration of Patent Document 1, the clamping position is limited, making it difficult to stabilize the posture and resulting in play corresponding to the gap between the shaft and the hole. Therefore, there is a problem that the positional relationship between the lens and the sensor deviates from the expected positional relationship, and the effect of improving image quality is weakened.

[0192] In contrast, according to the technology of this disclosure, since the surfaces of the V-shaped groove 1202 and the base 1300 facing the moving body 1200 are supported by three perfectly spherical members 1210a to 1210c, the moving body 1200 can be held without gaps. Furthermore, since the three perfectly spherical members 1210a to 1210c, which serve as supporting portions, are spaced apart from each other, the center of gravity of the moving body 1200 can be located on the inner side, and its posture can be stabilized. Moreover, according to the technology of this disclosure, the structure of the sensor module 1000 can be simplified by using the magnetic force generated by a permanent magnet as a clamping force.

[0193] Therefore, by applying the techniques of this disclosure, the super-resolution effect can be improved due to the attitude stability resulting from the suppression of the clearance associated with pixel shift driving. Furthermore, the magnetic force generated by the permanent magnet is used to press the moving body 1200, which can reduce the size of the sensor module 1000, increase the degree of freedom in its placement within the product, improve durability, and extend the product lifespan.

[0194] Furthermore, Patent Document 2 describes a pixel arrangement angle of 45 degrees relative to the image sensor. The directions, either horizontal or vertical, are used as examples of driving directions, but other driving directions are not described. Furthermore, regarding the effect of super-resolution, since the R, G, and B pixels in the Bayer arrangement are arranged differently, the effect varies in the R, G, and B colors depending on the driving direction and method.

[0195] In contrast, according to the technology of this disclosure, when a pixel shift is performed in a Bayer arrangement, the position of the G pixel does not overlap before and after the pixel shift. Therefore, the resolution of each of the R, G, and B colors in one direction can be improved.

[0196] Furthermore, the pixel shifting technique according to this disclosure, which involves shifting one pixel horizontally and two pixels vertically, has the advantage of excellent scalability, even though the effect of over-image shifting is equivalent to that of shifting one pixel horizontally. By performing pixel shifting using a drive signal with progressively changing signal levels, the spatial resolution bandwidth in the vertical direction can also be extended. Moreover, by increasing the number of steps for changing the signal level and increasing the number of steps for pixel shifting, super-resolution effects can be achieved in both the horizontal and vertical directions.

[0197] 3. Other implementation methods

[0198] In the embodiments described above, pixel shifting is performed by moving the image sensor 110 (imaging element 1100) to achieve super-resolution; however, this disclosure is not limited to this example. For instance, pixel shifting can be performed by moving the lens unit 1110 relative to the imaging element 1100.

[0199] Note that the effects described in this article are merely examples and not limitations, and other effects may occur.

[0200] Note that this technology can also be configured as follows. (1)

[0202] An imaging device, comprising:

[0203] A moving mechanism is configured to tilt an imaging element relative to a two-dimensional dot matrix in a plane perpendicular to the incident direction of the incident light, such that the amount of movement in a first direction along the two-dimensional dot matrix is ​​different from the amount of movement in a second direction different from the first direction. The imaging element includes a pixel array in which pixels configured to generate signals according to the incident light are arranged in a two-dimensional dot matrix, and each pixel in the predetermined arrangement is provided with a filter with different characteristics; and

[0204] The control unit is configured to perform drive control of the moving mechanism and imaging control of the imaging element, wherein,

[0205] Whenever the imaging element is moved by the moving mechanism, the control unit uses the imaging element to perform imaging. (2)

[0207] According to the imaging device described above (1), wherein,

[0208] The control unit performs drive control of the moving mechanism to move the imaging element to various divided positions obtained by dividing the period, in which the predetermined arrangement is reproduced by the movement performed by the moving mechanism. (3)

[0210] According to the imaging device described above (2), wherein,

[0211] The position is obtained by dividing the period by a power of two. (4)

[0213] According to the imaging device described in (2) or (3) above, wherein,

[0214] The control unit performs drive control on the moving mechanism to sequentially move the imaging element to each of the defined positions. (5)

[0216] According to the imaging device described in (2) or (3) above, wherein,

[0217] The control unit performs drive control on the moving mechanism to move the imaging element sequentially until it reaches the position with the maximum amount of movement among the divided positions, while sparsening the divided positions, and for each sparsed divided position, moves the imaging element sequentially from the position with the maximum amount of movement toward the position with the smaller amount of movement among the divided positions. (6)

[0219] According to any one of (1) to (5) described above, the imaging apparatus wherein,

[0220] The control unit moves the imaging element in sync with the frame period of the imaging process. (7)

[0222] The imaging apparatus according to any one of (1) to (6) described above, wherein,

[0223] The moving mechanism uses a piezoelectric element configured to expand and contract in one direction according to voltage to move the imaging element. (8)

[0225] According to the imaging device described above (7), wherein,

[0226] Mobile institutions include:

[0227] The base, to which the piezoelectric element is fixed, and

[0228] A movable body, configured to contact the end of the piezoelectric element in the expansion and contraction directions, is configured to move relative to the base according to the expansion and contraction of the piezoelectric element.

[0229] The imaging element is fixed to the moving body. (9)

[0231] According to the imaging device described above (8), wherein,

[0232] The base is provided with a first magnet having a magnetic force direction aligned with the expansion and contraction directions of the piezoelectric element, and

[0233] The moving body is provided with a first magnetic body facing the first magnet. (10)

[0235] According to the imaging apparatus described above in (8) or (9), wherein,

[0236] Mobile agencies also include:

[0237] Multiple perfect spheres are sandwiched between a first surface of the base and a second surface of the movable body. The first surface is a surface provided with piezoelectric elements, and the second surface faces the first surface and is parallel to the first surface. (11)

[0239] According to the imaging device described above (10), wherein,

[0240] The movable body has a V-shaped groove and a recess in its second surface. The V-shaped groove extends in the direction of movement of the movable body relative to the base.

[0241] The base includes two holes arranged in a manner corresponding to a V-shaped groove, and one hole arranged in a manner corresponding to a recess in the first surface.

[0242] Each of the holes provided in the first surface has a tapered surface with the opening in the first surface as its base.

[0243] At least two of the plurality of perfect spheres are sandwiched between a V-shaped groove and a plurality of holes arranged in a manner corresponding to the V-shaped groove, and

[0244] At least one of the plurality of perfect spheres, excluding the perfect sphere sandwiched between the V-shaped groove and the plurality of holes, is sandwiched between a recess and a hole provided in a manner corresponding to the recess. (12)

[0246] According to the imaging device described above (11), wherein,

[0247] Two holes and one hole are arranged in a positional relationship in which the center of gravity for the movement of the moving body and the driving range for moving the moving body are included within the triangle formed by the two holes and the one hole. (13)

[0249] According to any one of (10) to (12) described above, the imaging apparatus wherein,

[0250] The base is provided with a second magnet having a magnetic force direction aligned with the direction from the first surface toward the second surface, and

[0251] The moving body has a second magnetic body disposed on the second surface. (14)

[0253] According to any one of (10) to (13) described above, the imaging apparatus wherein,

[0254] Each of the multiple perfect spheres is a non-magnetic body. (15)

[0256] According to any one of (10) to (14) described above, the imaging apparatus wherein,

[0257] Each of the multiple perfect spheres is ceramic. (16)

[0259] According to any one of (1) to (15) described above, the imaging apparatus wherein,

[0260] The planned setup is a Bayer setup. (17)

[0262] An electronic device, comprising:

[0263] A moving mechanism is configured to move an imaging element at an angle relative to a two-dimensional dot matrix on a plane perpendicular to the incident direction of the incident light, such that the amount of movement along a first direction of the two-dimensional dot matrix is ​​different from the amount of movement in a second direction different from the first direction. The imaging element includes a pixel array in which pixels configured to output signals according to the incident light are arranged in a two-dimensional dot matrix, and each pixel in the predetermined arrangement is provided with a filter with different characteristics.

[0264] A control unit configured to perform drive control of the moving mechanism and imaging control of the imaging element; and

[0265] An imaging processing unit is configured to perform signal processing on signals output from an imaging element to generate image data, wherein...

[0266] Whenever the imaging element is moved by the moving mechanism, the control unit uses the imaging element to perform imaging.

Claims

1. An imaging device, comprising: A moving mechanism is configured to move an imaging element at an angle relative to a two-dimensional dot matrix on a plane perpendicular to the incident direction of the incident light, such that the amount of movement along a first direction of the two-dimensional dot matrix is ​​different from the amount of movement in a second direction different from the first direction. The imaging element includes a pixel array in which pixels configured to generate signals according to the incident light are arranged in the two-dimensional dot matrix, and each pixel in the predetermined arrangement is provided with a filter with different characteristics. as well as A control unit, configured to perform drive control of the moving mechanism and imaging control of the imaging element, wherein, Whenever the imaging element is moved by the moving mechanism, the control unit uses the imaging element to perform imaging.

2. The imaging device according to claim 1, wherein, The control unit performs the drive control of the moving mechanism to move the imaging element to various division positions obtained by dividing the period, in which the predetermined arrangement is reproduced by the movement performed by the moving mechanism.

3. The imaging device according to claim 2, wherein, The position is obtained by dividing the period by a power of two.

4. The imaging device according to claim 2, wherein, The control unit performs the drive control on the moving mechanism to sequentially move the imaging element to the respective divided positions.

5. The imaging device according to claim 2, wherein, The control unit performs the drive control on the moving mechanism to move the imaging element sequentially until it reaches the position with the maximum amount of movement among the divided positions, while sparsening the divided positions, and for each sparsed divided position, moving the imaging element sequentially from the position with the maximum amount of movement toward the position with the smaller amount of movement among the divided positions.

6. The imaging apparatus according to claim 1, wherein, The control unit moves the imaging element in sync with the frame period of the imaging performed by the imaging element.

7. The imaging apparatus according to claim 1, wherein, The moving mechanism uses a piezoelectric element configured to expand and contract in one direction according to voltage to move the imaging element.

8. The imaging apparatus according to claim 7, wherein, The moving mechanism includes: The base, to which the piezoelectric element is fixed, and A movable body, configured to contact the end of the piezoelectric element in the expansion and contraction directions, is configured to move relative to the base according to the expansion and contraction of the piezoelectric element. The imaging element is fixed to the moving body.

9. The imaging apparatus according to claim 8, wherein, The base is provided with a first magnet having a magnetic force direction aligned with the expansion and contraction directions of the piezoelectric element, and The movable body is provided with a first magnetic body facing the first magnet.

10. The imaging apparatus according to claim 8, wherein, The mobile mechanism also includes: Multiple perfect spheres are sandwiched between a first surface of the base and a second surface of the movable body. The first surface is the surface on which the piezoelectric element is disposed, and the second surface faces the first surface and is parallel to the first surface.

11. The imaging apparatus according to claim 10, wherein, The movable body has a V-shaped groove and a recess in the second surface, the V-shaped groove extending in the direction of movement of the movable body relative to the base. The base includes two holes in the first surface that correspond to the V-shaped groove, and one hole that corresponds to the recess. Each of the holes provided in the first surface has a tapered surface with the opening in the first surface as its base. At least two of the plurality of perfect spheres are sandwiched between the V-shaped groove and a plurality of holes arranged in a manner corresponding to the V-shaped groove, and At least one of the plurality of perfect spheres, excluding the perfect sphere sandwiched between the V-shaped groove and the plurality of holes, is sandwiched between the recess and the hole arranged in a manner corresponding to the recess.

12. The imaging apparatus according to claim 11, wherein, The two holes and the one hole are arranged in a positional relationship in which the center of gravity for the movement of the moving body and the driving range for moving the moving body are included within the triangle formed by the two holes and the one hole.

13. The imaging apparatus according to claim 10, wherein, The base is provided with a second magnet having a magnetic force direction aligned with the direction from the first surface toward the second surface, and The movable body has a second magnetic body disposed on the second surface.

14. The imaging apparatus according to claim 10, wherein, Each of the plurality of perfect spheres is a non-magnetic body.

15. The imaging apparatus according to claim 10, wherein, Each of the plurality of perfect spheres is ceramic.

16. The imaging apparatus according to claim 1, wherein, The predetermined arrangement is the Bayer arrangement.

17. An electronic device comprising: A moving mechanism is configured to move an imaging element at an angle relative to a two-dimensional dot matrix on a plane perpendicular to the incident direction of the incident light, such that the amount of movement along a first direction of the two-dimensional dot matrix is ​​different from the amount of movement in a second direction different from the first direction. The imaging element includes a pixel array in which pixels configured to output signals according to the incident light are arranged in the two-dimensional dot matrix, and each pixel in the predetermined arrangement is provided with a filter with different characteristics. A control unit configured to perform drive control of the moving mechanism and imaging control of the imaging element; as well as An imaging processing unit is configured to perform signal processing on a signal output from the imaging element to generate image data, wherein... Whenever the imaging element is moved by the moving mechanism, the control unit uses the imaging element to perform imaging.

Images