Imaging device and imaging unit

Abstract

An imaging device comprising: a solid-state imaging element having a pixel array in which a plurality of pixels are arranged; and an imaging device main body that holds the solid-state imaging element, wherein the plurality of pixels include first pixels whose pixel information is read out at a first frame rate and second pixels whose pixel information is read out at a second frame rate that is higher than the first frame rate; and wherein the imaging device main body outputs an image generated from the pixel information read out at the first frame rate, or a corrected image obtained by correcting the image using an optical flow derived from the pixel information read out at the second frame rate.

Description

Technical Field

[0001] The present invention relates to an imaging device and an imaging unit.

Background Art

[0002] Conventionally, an imaging device (such as a smartphone) capable of imaging an imaging object such as a person, an object, or a landscape has been known. This imaging device includes an imaging unit having an optical system such as a lens and a solid-state imaging device such as an image sensor (Japanese Patent Application Laid-Open No. 2023-1788).

[0003] When imaging an imaging object with this imaging device, blurring or defocus may occur in the obtained captured image due to the fast movement of the imaging object or the shake of the imaging device during imaging. In particular, when the frame rate is reduced (i.e., the exposure time during imaging is increased) due to low ambient light, etc., the blurring or defocus in the captured image becomes prominent.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] Therefore, an object of the present invention is to provide an imaging device and an imaging unit that can clearly image a moving imaging object.

Means for Solving the Problems

[0006] The imaging device according to the present invention includes a solid-state imaging device having a pixel array in which a plurality of pixels are arranged in a matrix, and an imaging device body having an optical system. The solid-state image sensor is mounted on the imaging device body such that light passing through the optical system is incident on the pixel array. The aforementioned plurality of pixels are A plurality of first pixels whose pixel information is read at a first frame rate when the light is incident on the pixel array, The pixel array includes a plurality of second pixels, the pixel information of which is read at a second frame rate higher than the first frame rate when the light is incident on the pixel array, The imaging device outputs an image generated from pixel information read out at the first frame rate, or a corrected image obtained by correcting the image generated from pixel information read out at the first frame rate using an optical flow derived from pixel information read out at the second frame rate.

[0007] In the aforementioned imaging device, The frame rate at which the pixel information is read from the second pixel may be switched between the second frame rate and the first frame rate.

[0008] Furthermore, in the imaging device, The plurality of second pixels may be arranged in the pixel array with gaps between them.

[0009] Furthermore, in the imaging device, The number of first pixels may be greater than the number of second pixels.

[0010] Furthermore, in the imaging device, A first A / D converter to which the pixel information read from the first pixel is input, The system includes a second A / D converter to which the pixel information read from the second pixel is input, The resolution of the second A / D converter may be smaller than the resolution of the first A / D converter.

[0011] Furthermore, in the imaging device, The image sensor has a color filter in which red, green, and blue are arranged in a predetermined pattern and superimposed on the pixel array. The second pixel may be positioned in the pixel array at a location corresponding to the planned placement of red or blue in the pattern of the color filter.

[0012] The portion of the color filter corresponding to the second pixel may be colorless and transparent.

[0013] Furthermore, the imaging unit according to the present invention is A solid-state image sensor having a pixel array in which multiple pixels are arranged in a matrix, The system includes an image generation unit that generates an image based on pixel information from the plurality of pixels, The aforementioned plurality of pixels are Multiple first pixels whose pixel information is read at a first frame rate when light is incident on the aforementioned pixel array, The pixel array includes a plurality of second pixels, the pixel information of which is read at a second frame rate higher than the first frame rate when the light is incident on the pixel array, The image generation unit generates and outputs an image from the pixel information read at the first frame rate, or corrects the image generated from the pixel information read at the first frame rate using an optical flow derived from the pixel information read at the second frame rate, and outputs the corrected image.

[0014] In the aforementioned imaging unit, The image generation unit may reconstruct the pixel information of the missing coordinates of the first pixels in the pixel array based on the pixel information read from the plurality of second pixels, and generate an image based on the reconstructed pixel information and the pixel information read from each first pixel.

[0015] Furthermore, the imaging method according to the present invention is A step of reading pixel information when light is incident on a pixel array at a first frame rate, wherein a plurality of pixels are arranged in a matrix in the pixel array, and the plurality of pixels include a plurality of first pixels that read pixel information when light is incident on the pixel array at the first frame rate, and a plurality of second pixels that read pixel information when the light is incident on the pixel array at a second frame rate higher than the first frame rate; Output an image generated from the pixel information read in the first frame, Or, Reading the pixel information when the light is incident on the pixel array at the second frame rate, deriving an optical flow from the pixel information read at the second frame rate, and correcting the image generated from the pixel information read in the first frame by the optical flow and outputting the corrected image.

[0016] The method includes: When outputting an image generated from the pixel information read at the first frame rate, Restoring pixel information at the missing coordinates of the first pixels in the pixel array based on the pixel information read from the plurality of second pixels, and generating the image based on the restored pixel information and the pixel information read in the first frame.

Brief Description of the Drawings

[0017] [Figure 1] FIG. 1 is a functional block diagram of a smartphone according to this embodiment. [Figure 2] FIG. 2 is a view of the smartphone seen from the side where the optical system is arranged. [Figure 3] FIG. 3 is a diagram for explaining the configuration of a solid-state imaging device included in the smartphone. [Figure 4] FIG. 4 is a diagram for explaining the configuration of a color filter and a pixel array in the solid-state imaging device.

Modes for Carrying Out the Invention

[0018] The imaging device described herein is A solid-state image sensor having a pixel array in which multiple pixels are arranged in a matrix, The imaging device comprises an imaging device body having an optical system, The solid-state image sensor is mounted on the imaging device body such that light passing through the optical system is incident on the pixel array. The aforementioned plurality of pixels are A plurality of first pixels whose pixel information is read at a first frame rate when the light is incident on the pixel array, The pixel array includes a plurality of second pixels, the pixel information of which is read at a second frame rate higher than the first frame rate when the light is incident on the pixel array, The imaging device outputs an image generated from pixel information read out at the first frame rate, or a corrected image obtained by correcting the image generated from pixel information read out at the first frame rate using an optical flow derived from pixel information read out at the second frame rate.

[0019] With this configuration, an image of the object to be captured is generated from the pixel information of the first of multiple pixels of a single solid-state image sensor, and an optical flow is derived from the pixel information of the second pixel. As a result, even if blurring or out-of-focus images occur in the image generated from the pixel information of the first pixel due to the movement of the object to be captured or the shaking of the imaging device, the image is corrected by the optical flow, eliminating or suppressing the blurring or out-of-focus images, and as a result, a clear image (corrected image) is output (displayed, etc.).

[0020] Furthermore, if the first pixel is continuously capturing images at a certain frame rate for video recording, it is possible to generate video information exceeding the original frame rate of the first pixel using optical flow.

[0021] Furthermore, in the imaging device of this embodiment, The frame rate at which the pixel information is read from the second pixel may be switched between the second frame rate and the first frame rate.

[0022] With this configuration, in addition to the pixel information from the first pixel, the pixel information read from the second pixel at the first frame rate can also be used to generate an image, thereby further improving the image quality of the generated image.

[0023] Furthermore, in the imaging device of this embodiment, The plurality of second pixels may be arranged in the pixel array with gaps between them.

[0024] By arranging the second pixels with spacing between them in this way, pixel information for deriving optical flow can be obtained from the entire pixel array, thereby further improving the image quality of the corrected image.

[0025] Furthermore, in the imaging device of this embodiment, The number of first pixels may be greater than the number of second pixels.

[0026] With this configuration, the number of pixels necessary to generate an image is ensured, and even if a second pixel (a pixel for deriving the optical flow) is placed in the pixel array, the degradation of the image quality of the generated image is suppressed.

[0027] Furthermore, the imaging device of this embodiment is A first A / D converter to which the pixel information read from the first pixel is input, The system includes a second A / D converter to which the pixel information read from the second pixel is input, The resolution of the second A / D converter may be smaller than the resolution of the first A / D converter.

[0028] By reducing the resolution in this way, it becomes easier to ensure the processing speed of the second A / D converter, which allows the pixel information read at the second frame rate (high frame rate) to be processed effectively.

[0029] Furthermore, in the imaging device of this embodiment, The solid-state image sensor has a color filter in which red, green, and blue are arranged in a predetermined pattern and superimposed on the pixel array. The second pixel may be positioned in the pixel array at a location corresponding to the planned placement of red or blue in the pattern of the color filter.

[0030] Humans are sensitive to changes in green brightness (resolution), so by ensuring a sufficient number of first pixels in the green filter as shown in the above configuration, even if the number of pixels used to generate the image (first pixels) decreases due to the placement of second pixels in the pixel array, the degradation of the image quality of the generated image can be suppressed.

[0031] in this case, The portion of the color filter corresponding to the second pixel may be colorless and transparent.

[0032] This configuration makes it possible to suppress the decrease in the amount of light received by the second pixel caused by the color filter.

[0033] Furthermore, the imaging unit according to this embodiment is A solid-state image sensor having a pixel array in which multiple pixels are arranged in a matrix, The system includes an image generation unit that generates an image based on pixel information from the plurality of pixels, The aforementioned plurality of pixels are Multiple first pixels whose pixel information is read at a first frame rate when light is incident on the aforementioned pixel array, The pixel array includes a plurality of second pixels, the pixel information of which is read at a second frame rate higher than the first frame rate when the light is incident on the pixel array, The image generation unit generates and outputs an image from the pixel information read at the first frame rate, or corrects the image generated from the pixel information read at the first frame rate using an optical flow derived from the pixel information read at the second frame rate, and outputs the corrected image (corrected image).

[0034] With this configuration, an image of the object to be captured is generated from the pixel information of the first of multiple pixels of a single solid-state image sensor, and an optical flow is derived from the pixel information of the second pixel. As a result, even if blurring or out-of-focus images occur in the image generated from the pixel information of the first pixel due to the movement of the object to be captured or the shaking of the imaging device, the optical flow corrects the image, eliminating or suppressing the blurring or out-of-focus images, and consequently outputting a clear image (corrected image).

[0035] Furthermore, if the first pixel is continuously capturing images at a certain frame rate for video recording, optical flow can be used to generate and output video information exceeding the original frame rate of the first pixel.

[0036] Furthermore, in the imaging unit of this embodiment, The image generation unit may reconstruct the pixel information of the missing coordinates of the first pixels in the pixel array based on the pixel information read from the plurality of second pixels, and generate an image based on the reconstructed pixel information and the pixel information read from each first pixel.

[0037] In this way, by reconstructing (predicting) the pixel information of the missing coordinates of the first pixel at the position where the second pixel is located in the pixel array—that is, the pixel information that would have been obtained (read out) from the first pixel if it had been located at the position of the second pixel—and using this information to generate the image, the image quality of the generated image is improved.

[0038] Furthermore, the imaging method according to this embodiment is A step of reading out pixel information at a first frame rate when light is incident on a pixel array, wherein the pixel array has a plurality of pixels arranged in a matrix, and the plurality of pixels include a plurality of first pixels whose pixel information is read out at a first frame rate when light is incident on the pixel array, and a plurality of second pixels whose pixel information is read out at a second frame rate higher than the first frame rate when light is incident on the pixel array, Output an image generated from the pixel information read in the first frame, or The process includes the steps of: reading out pixel information at the second frame rate when the light is incident on the pixel array; deriving an optical flow from the pixel information read out at the second frame rate; correcting the image generated from the pixel information read out in the first frame using the optical flow; and outputting the corrected image (corrected image).

[0039] According to this method, an image of the subject to be captured is generated from the pixel information of the first of multiple pixels, and an optical flow is derived from the pixel information of the second pixel. As a result, even if blurring or out-of-focus images occur in the image generated from the pixel information of the first pixel due to the movement of the subject to be captured or the shaking of the imaging device, the optical flow corrects the image, eliminating or suppressing the blurring or out-of-focus images, and consequently outputting a clear image (corrected image).

[0040] Furthermore, if the first pixel is continuously capturing images at a certain frame rate for video recording, optical flow can be used to generate and output video information exceeding the original frame rate of the first pixel.

[0041] Furthermore, the method of this embodiment is When outputting an image generated from pixel information read at the aforementioned first frame rate, The pixel information of the missing coordinates of the first pixel in the pixel array may be restored based on the pixel information read from the plurality of second pixels, and the image may be generated based on the restored pixel information and the pixel information read in the first frame.

[0042] In this way, by reconstructing (predicting) the pixel information of the missing coordinates of the first pixel at the position where the second pixel is located in the pixel array—that is, the pixel information that would have been obtained (read out) from the first pixel if it had been located at the position of the second pixel—and using this information to generate the image, the image quality of the generated image is improved.

[0043] As described above, according to this embodiment, it is possible to provide an imaging device and an imaging unit that can clearly image moving objects.

[0044] One embodiment of the present invention will be described below with reference to the attached drawings.

[0045] The imaging device is capable of capturing still images and videos, and can be a digital camera, smartphone, tablet device, etc. In this embodiment, the imaging device 1 is a smartphone, as shown in Figures 1 and 2.

[0046] Specifically, as shown in Figure 3, this smartphone (imaging device) 1 comprises a solid-state image sensor 2 having a pixel array 21 and a smartphone body (imaging device body) 3 having an optical system 31 composed of at least one optical element such as a lens. The solid-state image sensor 2 is mounted on the smartphone body 3 so that light passing through the optical system 31 is incident on the pixel array 21. The smartphone 1 also includes a control unit 4 connected to the solid-state image sensor 2. The smartphone 1 of this embodiment also includes a non-volatile memory 101, a working memory 102, an operation unit 103, a display unit 104, a recording medium 105, a connection unit 106, a short-range wireless communication unit 107, a public network connection unit 108, a microphone 109, and a speaker 110.

[0047] Under the control of the control unit 4, the smartphone 1 converts the image of the subject (object) formed by the optical system 31 of the smartphone body 3 into an electrical signal, performs noise reduction processing, etc., and outputs the digital data as an output image (image data) (for example, displayed on the display unit 104).

[0048] The solid-state image sensor 2 is an element for imaging an object, more specifically, an element that converts the image of the object formed through the optical system 31 into an electrical signal. In this embodiment, the solid-state image sensor 2 is, for example, a CMOS image sensor.

[0049] As shown in Figure 4, this solid-state image sensor 2 includes a pixel array 21, a color filter 25 superimposed on the pixel array 21, and an A / D conversion unit 26 that performs A / D conversion on the signals (pixel information) read from each pixel 22.

[0050] The color filter 25 restricts the wavelength of light incident on each pixel 22, and filters that allow red, green, and blue wavelengths to pass through are arranged in a predetermined pattern. In this embodiment, the arrangement pattern of the red, green, and blue filters in the color filter 25 is a Bayer array. However, the arrangement pattern is not limited to a Bayer array of red, green, and blue, and may be other colors or other arrangement patterns.

[0051] In the pixel array 21, multiple pixels 22, each having a photoelectric conversion element FD, are arranged in a matrix (see Figure 3). That is, the pixel array 21 is composed of multiple pixels 22 arranged in a matrix. Note that Figure 3 shows a part of the pixel array 21, and in the pixel array 21, the arrangement shown in Figure 3 is repeated in the row direction (left-right direction in Figure 3) and the column direction (up-down direction in Figure 3). In addition, each pixel 22 in Figures 3 and 4 has a pattern such as a dot to indicate the difference in color of the color filter 25 (see Figure 4).

[0052] These multiple pixels 22 include multiple first pixels 23 whose pixel information (voltage signals corresponding to the signal charge generated by the photoelectric conversion element FD of each pixel 22) is read out at a first frame rate when light passing through the optical system 31 is incident on the pixel array 21, and multiple second pixels 24 whose pixel information is read out at a second frame rate higher than the first frame rate when light passing through the optical system 31 is incident on the pixel array 21. The first pixels 23 and second pixels 24 in this embodiment have the same configuration, differing only in the frame rate at which the pixel information is read out.

[0053] In this embodiment, the first frame rate is, for example, 60 fps, and the second frame rate is, for example, 1000 fps. The first frame rate may be the readout rate of the RGB pixels of the CMOS image sensor, and is expressed in units of times per second. The second frame rate may be the readout rate of the EVS pixels of the CMOS image sensor, and is similarly expressed in units of times per second. Typically, the second frame rate is much larger than the first frame rate. However, the frame rate when pixel information is read out from each pixel 23, 24 is not limited to these values.

[0054] Furthermore, in the pixel array 21 of this embodiment, there are more first pixels 23 than second pixels 24. For example, the ratio of the number of first pixels 23 to the number of second pixels 24 in the pixel array 21 is 8:1, 16:1, 32:1, etc.

[0055] Multiple second pixels 24 are arranged in the pixel array 21 with spacing between them (see Figure 4). Specifically, the multiple second pixels 24 are arranged approximately evenly throughout the entire pixel array 21 such that the spacing between adjacent second pixels 24 is roughly the same.

[0056] Furthermore, each second pixel 24 is positioned in the pixel array 21 at a location corresponding to the planned placement of red or blue in the arrangement pattern of the color filter 25 (in this embodiment, a Bayer array). Note that the portion of the color filter 25 corresponding to the second pixel 24 is colorless and transparent. In other words, in the color filter 25 of this embodiment, the number of red or blue filters is reduced without reducing the number of green filters in the Bayer array arrangement, and the positions where the red or blue filters were reduced are made colorless and transparent.

[0057] The A / D conversion unit 26 converts pixel information (analog signals) from the pixel array 21 into digital signals and outputs them to the next stage. The A / D conversion unit 26 in this embodiment has a plurality of first A / D converters 261 that receive pixel information read from the first pixel 23, and a plurality of second A / D converters 262 that receive pixel information read from the second pixel 24. These first A / D converters 261 and second A / D converters 262 are arranged for each column (column of pixels 22) of the pixel array 21 in which the pixels 22 are arranged in a matrix.

[0058] Each first A / D converter 261 is connected to each first pixel 23 in the column direction of the pixel array 21 via a first signal line 27a, and each second A / D converter 262 is connected to each second pixel 24 in the column direction of the pixel array 21 via a second signal line 27b. In the A / D conversion unit 26 of this embodiment, the resolution of the second A / D converter 262 is smaller than the resolution of the first A / D converter 261. For example, the resolution of the second A / D converter 262 in this embodiment is 6 bits, and the resolution of the first A / D converter 261 in this embodiment is 10 bits. In Figure 3, the lines connected to each first pixel 23 in the row direction are first control lines 28a, and the lines connected to each second pixel 24 in the row direction are second control lines 28b. These control lines 28a and 28b are wiring for controlling each pixel 23 and 24, such as resetting each pixel 23 and 24 and selecting the row from which to read pixel information (signals).

[0059] The control unit 4 controls each part of the smartphone 1 according to the input signals and programs. The control unit 4 also generates an image (image for external output) from the signals (pixel information) output from the solid-state image sensor 2 and outputs it to the display unit 104. It should be noted that the control unit that controls each part of the smartphone 1 and the control unit 4 that performs control related to image capture may be provided separately.

[0060] The control unit 4 has an arithmetic processing unit (image generation unit) 41 that is directly or indirectly connected to the solid-state image sensor 2. This arithmetic processing unit 41 derives optical flow from pixel information read from the second pixel 24, generates an image from pixel information read from the first pixel 23, and corrects the generated image using the derived optical flow. Functionally, the arithmetic processing unit 41 of this embodiment includes an image generation unit that generates an image from pixel information read from the first pixel 23, an optical flow calculation unit that derives optical flow from pixel information read from the second pixel 24, and a correction calculation unit that corrects the image using the optical flow.

[0061] Here, optical flow refers to the optical movement of each point between two temporally consecutive image frames, which can be observed on a digital image by projecting the movement of the object being captured onto the image. For example, it is the vectorization of the difference between feature points (moving points) between two temporally adjacent image frames in a video. In the smartphone 1 of this embodiment, the pixel array 21 of the solid-state image sensor 2 is used to perform multiple imaging cycles by increasing the frame rate (second frame rate) when reading out the pixel information of the second pixel 24 (first frame rate) above the frame rate (first frame rate) when reading out the pixel information of the first pixel 23. From each of the multiple images acquired in this imaging cycle (images captured by the second pixel 24), the processing unit 41 calculates the vectors (direction of movement and distance) of each moving point, thereby deriving the optical flow.

[0062] Furthermore, when the arithmetic processing unit 41 derives (calculates) the optical flow from the pixel information read from the second pixel 24, it creates an event signal quantized into three values ​​of +1 / 0 / -1 from the difference of a continuous image (an image based on the pixel information read from each second pixel 24), reduces the data transmission bandwidth, and then derives the optical flow using the created event signal. Here, +1 means that the brightness has increased by more than a certain percentage, 0 means that the change in brightness is within a certain range, and -1 means that the brightness has decreased by more than a certain amount. In addition, when the arithmetic processing unit 41 creates an event signal from the pixel information read from the second pixel 24, it may also create the event signal based on the brightness signal (Y). Moreover, if there is sufficient data transmission bandwidth, the arithmetic processing unit 41 may also derive the optical flow by directly calculating it from a brightness signal with gradation. Furthermore, the method of deriving (calculating) the optical flow is not limited to the method described above. Various conventional methods (for example, the gradient method or the Lucas-Kanade method) may be used to derive the optical flow.

[0063] The non-volatile memory 101 is an electrically erasable and recordable non-volatile memory. In this embodiment, the non-volatile memory 101 stores the OS (operating system), which is the basic software executed by the control unit 4, and applications that work in cooperation with this OS to realize advanced functions.

[0064] The working memory 102 is used as the image display memory for the display unit 104 and as the working area for the control unit 4, etc.

[0065] The operation unit 103 is used by the user to input instructions for the smartphone 1. In this embodiment, the operation unit 103 includes a power button for instructing the smartphone 1 to turn on / off, and a touch panel formed on the display unit 104.

[0066] The display unit 104 displays (outputs to an external source) the captured image (image data) and displays text for operation, etc.

[0067] The recording medium 105 records the image data (captured image) output from the control unit 4.

[0068] The connection unit 106 is an interface for connecting to an external device. The smartphone 1 exchanges data with the external device via this connection unit 106.

[0069] The short-range wireless communication unit 107 is a communication unit for performing short-range wireless communication. The short-range wireless communication unit 107 consists of an antenna for wireless communication and a modulation / demodulation circuit and communication controller for processing wireless signals.

[0070] The public network connection unit 108 is an interface for performing public wireless communication. Through this public network connection unit 108, the smartphone 1 communicates with other devices for making calls. At this time, the control unit 4 realizes the call by inputting and outputting audio signals via the microphone 109 and speaker 110. In this embodiment, the public network connection unit 108 is an antenna, and the control unit 4 connects to the public network via this antenna.

[0071] With the smartphone 1 configured as described above, blur and out-of-focus images can be corrected in the following way.

[0072] The target object is captured by the smartphone 1. At this time, the solid-state image sensor 2 reads out pixel information (first image) from each first pixel 23 at 60 frames per second (fps), while reading out pixel information (second image) from each second pixel 24 at 1000 fps. Specifically, in Figure 3, the multiple first pixels 23 corresponding to the first control line 28a are scanned sequentially from top to bottom 60 times per second, while independently of this, the multiple second pixels 24 corresponding to the second control line 28b are scanned sequentially from top to bottom 1000 times per second.

[0073] Furthermore, if not all of the multiple second A / D converters 262 arranged in the row direction in Figure 3 are occupied by pixel information, multiple second control lines 28b may be activated simultaneously so that pixel information is input to each of the multiple second A / D converters 262 from the second pixels 24 connected via each second signal line 27b.

[0074] When multiple second images (images based on pixel information read from each second pixel 24) are read from each second pixel 24 in the pixel array 21, the arithmetic processing unit 41 derives an optical flow and uses this derived optical flow to correct the corresponding first image (image based on pixel information read from each first pixel 23). This corrects blur and blur caused by the movement of the object being captured in the first image, resulting in a clear image (corrected image).

[0075] Furthermore, in this corrected image, the control unit 4 of this embodiment determines the reliability of the captured image corrected using optical flow, and if the reliability is low, it outputs the captured image without correction (an image generated from the pixel information read from the first pixel 23).

[0076] This confidence level is determined using, for example, object recognition by AI. In this case, for instance, a low confidence level is determined when a human face is not recognized as a human face.

[0077] Furthermore, a second image based on the pixel information read from the second pixel 24 may be used to determine the reliability. Specifically, the second image output from the second pixel 24 has low resolution and a poor signal-to-noise ratio, but it contains unblurred information, so the reliability of the correction result is determined based on this second image. In addition, the reliability of the first image corrected using optical flow may be determined, and if the reliability is low, the correction may be performed again using the second image output from the second pixel 24. Specifically, the second image output from the second pixel 24 has low resolution and a poor signal-to-noise ratio, but it contains unblurred information, so the image information from the first pixel 23 is replaced or blended based on this image information.

[0078] The smartphone (imaging device) 1 described above comprises a solid-state image sensor 2 having a pixel array 21 in which a plurality of pixels 22 are arranged in a matrix, and a smartphone body (imaging device body) 3 having an optical system 31. The solid-state image sensor 2 is mounted on the smartphone body 3 so that light passing through the optical system 31 is incident on the pixel array 21. The plurality of pixels 22 include a plurality of first pixels 23 whose pixel information is read at a first frame rate when light is incident on the pixel array 21, and a plurality of second pixels 24 whose pixel information is read at a second frame rate higher than the first frame rate when light is incident on the pixel array 21. The smartphone body 3 outputs an image generated from the pixel information read at the first frame rate, or a corrected image obtained by correcting the image generated from the pixel information read at the first frame rate with optical flow derived from the pixel information read at the second frame rate.

[0079] With this configuration, an image of the object to be captured is generated from the pixel information of the first pixel 23 among the multiple pixels 22 of a single solid-state image sensor 2, and optical flow is derived from the pixel information of the second pixel 24. As a result, even if blurring or out-of-focus images occur in the image generated from the pixel information of the first pixel 23 due to the movement of the object to be captured or the shaking of the imaging device, the image is corrected by the optical flow, eliminating or suppressing the blurring or out-of-focus images, and as a result, a clear image (corrected image) is output.

[0080] Furthermore, if the first pixel 23 is continuously capturing images at a certain frame rate for video capture, it is possible to generate video information exceeding the original frame rate of the first pixel 23 using optical flow.

[0081] Furthermore, in the smartphone 1 of this embodiment, the multiple second pixels 24 are arranged at intervals from each other in the pixel array 21.

[0082] By arranging the second pixels 24 with spacing between them in this way, pixel information for deriving optical flow can be obtained from the entire pixel array 21, further improving the image quality of the corrected image.

[0083] Furthermore, in the smartphone 1 of this embodiment, there are more first pixels 23 than second pixels 24. Therefore, the number of pixels (number of first pixels 23) necessary for generating an image is secured, and even if second pixels (pixels for deriving optical flow) 24 are arranged in the pixel array 21, a decrease in the image quality of the generated image is prevented.

[0084] Furthermore, the smartphone 1 of this embodiment includes a first A / D converter that receives pixel information read from a first pixel 23, and a second A / D converter that receives pixel information read from a second pixel 24, wherein the resolution of the second A / D converter is smaller than the resolution of the first A / D converter.

[0085] With this configuration, it is easier to ensure the processing speed of the second A / D converter, thereby allowing the pixel information read at the second frame rate to be processed favorably.

[0086] Furthermore, in the smartphone 1 of this embodiment, the solid-state image sensor 2 has a color filter 25 in which red, green, and blue are arranged in a predetermined pattern and superimposed on the pixel array 21, and the second pixel 24 is positioned in the pixel array 21 at a position corresponding to the planned position of red or blue in the pattern of the color filter 25.

[0087] Since humans are sensitive to changes in green brightness (resolution), by ensuring a sufficient number of first pixels 23 with a green filter superimposed as in the above configuration, even if the number of pixels (first pixels) 23 used to generate the image decreases due to the placement of second pixels 24 in the pixel array 21, the degradation of the generated image quality can be suppressed.

[0088] Furthermore, in the smartphone 1 of this embodiment, the portion of the color filter 25 corresponding to the second pixel 24 is colorless and transparent.

[0089] With this configuration, the decrease in the amount of light received by the second pixel 24 caused by the color filter 25 can be suppressed.

[0090] It should be noted that the imaging apparatus of the present invention is not limited to the embodiments described above, and various modifications can be made without departing from the spirit of the invention. For example, the configuration of one embodiment can be added to the configuration of another embodiment, and a part of the configuration of one embodiment can be replaced with the configuration of another embodiment. Furthermore, a part of the configuration of one embodiment can be deleted.

[0091] The imaging device 1 in the above embodiment is a smartphone, but is not limited to this configuration. The imaging device 1 may be other devices (devices with imaging capabilities) such as a digital camera or a tablet device (mobile terminal).

[0092] Furthermore, although the imaging device 1 in the above embodiment includes one imaging unit (optical system 31 and a solid-state image sensor 2 corresponding to the optical system 31), it is not limited to this configuration. The imaging device 1 may include multiple imaging units. Specifically, for example, the imaging device 1 may include a telephoto imaging unit capable of telephoto imaging, a wide-angle imaging unit capable of wide-angle imaging, and an ultra-wide-angle imaging unit capable of ultra-wide-angle imaging. In this case, the focal length of the telephoto imaging unit is 80 mm or more in 35 mm film equivalent, the focal length of the wide-angle imaging unit is 24 mm in 35 mm film equivalent, and the focal length of the ultra-wide-angle imaging unit is 14 mm or less in 35 mm film equivalent.

[0093] In this case, the pixel array 21 of the solid-state image sensor 2 in each imaging unit may have a first pixel 23 and a second pixel 24, and each imaging unit may perform imaging at a first frame rate with the first pixel 23 and imaging at a second frame rate with the second pixel 24. That is, each imaging unit may perform image generation and optical flow derivation.

[0094] Alternatively, for example, the solid-state image sensor 2 of the imaging unit with the widest field of view among the multiple imaging units (the ultra-wide-angle imaging unit in the above example) may have a first pixel 23 and a second pixel 24, while the solid-state image sensors 2 of the remaining imaging units (the wide-angle imaging unit and the telephoto imaging unit in the above example) may have only a first pixel 23. In this case, regardless of which imaging unit is used for imaging, an optical flow is derived by imaging with the second pixel 24 of the ultra-wide-angle imaging unit, and this optical flow corrects the image captured by the first pixel 23 of each imaging unit.

[0095] Furthermore, while the imaging device 1 in the above embodiment derived optical flow for each image and determined whether or not to correct the image (the image generated from the pixel information of the first pixel 23) using the derived optical flow based on its reliability, the device is not limited to this configuration. The device may also be configured in which the imager or the like manually selects whether or not to derive optical flow. Alternatively, for example, the device may be configured to automatically switch to deriving optical flow when the accumulation time of incident light in the solid-state image sensor 2 for normal imaging is 1 / 1000 of a second or longer.

[0096] Furthermore, in the imaging device 1 of the above embodiment, pixel information is read from the second pixel 24 at the second frame rate, but the device is not limited to this configuration. The imaging device 1 may also be configured to allow switching between the second frame rate and the first frame rate for reading pixel information from the second pixel 24. If the image generated by the pixel information from the first pixel 23 does not exhibit blurring or out-of-focus, the image quality of the generated image can be further improved by reading the pixel information from the second pixel 24 at the first frame rate and using it to generate the image.

[0097] Furthermore, although the arithmetic processing unit 41 of the imaging device 1 in the above embodiment is included in the control unit 4 that controls the imaging device (smartphone) 1, the configuration is not limited to this. The solid-state image sensor 2 may also have an arithmetic processing unit 41. Alternatively, the solid-state image sensor 2 and the arithmetic processing unit (image generation unit) 41 may constitute an imaging unit, and this imaging unit may be mounted on the imaging device 1.

[0098] In this imaging unit, the image generation unit may reconstruct (predict) the pixel information of the missing coordinates of the first pixels 23 in the pixel array 21 based on the pixel information read from a plurality of second pixels 24, that is, the pixel information that would be obtained (read) from the first pixels 23 if the first pixels 23 were located at the position of the second pixels 24, and generate an image based on the reconstructed pixel information and the pixel information read from each first pixel 23. In this case, the reconstruction may be performed in conjunction with correction by optical flow during image generation by the image generation unit, or the reconstruction may be performed with correction by optical flow turned off. The reconstruction (reconstruction of pixel information at the missing coordinates (calculation, prediction, etc.)) may be calculated by calculation, etc., based on the pixel information read from the first pixels 23 around the coordinates, or it may be predicted by AI, etc.

[0099] In this way, by restoring (predicting, etc.) the pixel information of the missing coordinates of the first pixel 23 at the position where the second pixel 24 is located in the pixel array 21, that is, the pixel information that would have been obtained (read out) from the first pixel 23 if it had been located at the position where the second pixel 24 is located, and using this information to generate the image, the image quality of the generated image is further improved.

[0100] Furthermore, in the imaging device 1 of the above embodiment, the first pixel 23 and the second pixel 24 of the solid-state image sensor 2 have the same configuration and operation, but the device is not limited to this configuration. The first pixel 23 and the second pixel 24 may have different configurations and operations. As a characteristic required of the second pixel 24 is to output a large signal with a short exposure time, the second pixel 24 may, for example, be designed or set to have a conversion gain for converting signal charge into voltage that is larger than the first gain, which is the conversion gain of the first pixel 23, but the device is not limited to this configuration either.

[0101] In order to express the present invention, the embodiments have been adequately and sufficiently described above with reference to the drawings. However, those skilled in the art should recognize that it is easy to modify and / or improve upon the above embodiments. Therefore, unless such modifications or improvements implemented by those skilled in the art deviate from the scope of the claims, such modifications or improvements shall be considered to be included within the scope of the claims. [Explanation of symbols]

[0102] 1...Smartphone (imaging device), 2...Solid-state image sensor, 21...Pixel array, 22...Pixel, 23...First pixel, 24...Second pixel, 25...Color filter, 26...A / D conversion unit, 261...First A / D converter, 262...Second A / D converter, 27a...First signal line, 27b...Second signal line, 28a...First control line, 28b...Second control line, 3...Smartphone body, 31...Optical system, 4...Control unit, 41...Calculation processing unit, 101...Non-volatile memory, 102...Working memory, 103...Operation unit, 104...Display unit, 105...Recording medium, 106...Connection unit, 107...Short-range wireless communication unit, 108...Public network connection unit, 109...Microphone, 110...Speaker, FD...Photoelectric conversion element

Claims

1. A solid-state image sensor having a pixel array in which multiple pixels are arranged in a matrix, The imaging device comprises an imaging device body having an optical system, The solid-state image sensor is mounted on the imaging device body such that light passing through the optical system is incident on the pixel array. The aforementioned plurality of pixels are A plurality of first pixels whose pixel information is read at a first frame rate when the light is incident on the pixel array, The pixel array includes a plurality of second pixels, the pixel information of which is read at a second frame rate higher than the first frame rate when the light is incident on the pixel array, The plurality of second pixels are arranged in the pixel array with intervals between them. The imaging device outputs an image generated from pixel information read out at the first frame rate, or a corrected image obtained by correcting the image generated from pixel information read out at the first frame rate with an optical flow derived from pixel information read out at the second frame rate.

2. The imaging apparatus according to claim 1, wherein the frame rate for reading the pixel information from the second pixel can be switched between the second frame rate and the first frame rate.

3. The optical flow is derived as follows: an event signal is created from the difference between consecutive images, quantized into three values: +1, 0, and -1; the data transmission bandwidth is further reduced; the optical flow is derived using the created event signal; the image is based on pixel information read out at the second frame rate; +1 means that the brightness increases by a certain percentage or more; 0 means that the change in brightness is within a certain range; and -1 means that the brightness decreases by a certain percentage or more, according to claim 1.

4. The imaging apparatus according to any one of claims 1 to 3, wherein the first pixel is greater than the second pixel.

5. A first A / D converter to which the pixel information read from the first pixel is input, The system includes a second A / D converter to which the pixel information read from the second pixel is input, The imaging apparatus according to any one of claims 1 to 3, wherein the resolution of the second A / D converter is smaller than the resolution of the first A / D converter.

6. The solid-state image sensor has a color filter in which red, green, and blue are arranged in a predetermined pattern and superimposed on the pixel array. The imaging apparatus according to any one of claims 1 to 3, wherein the second pixel is positioned in the pixel array at a position corresponding to the planned placement position of red or blue in the pattern of the color filter.

7. The imaging apparatus according to claim 6, wherein the portion of the color filter corresponding to the second pixel is colorless and transparent.

8. A solid-state image sensor having a pixel array in which multiple pixels are arranged in a matrix, The system includes an image generation unit that generates an image based on pixel information from the plurality of pixels, The aforementioned plurality of pixels are Multiple first pixels whose pixel information is read at a first frame rate when light is incident on the aforementioned pixel array, The pixel array includes a plurality of second pixels, the pixel information of which is read at a second frame rate higher than the first frame rate when the light is incident on the pixel array, The plurality of second pixels are arranged in the pixel array with intervals between them. The image generation unit generates and outputs an image from pixel information read at the first frame rate, or corrects the image generated from the pixel information read at the first frame rate using an optical flow derived from the pixel information read at the second frame rate, and outputs the corrected image.

9. The imaging unit according to claim 8, wherein the image generation unit restores pixel information for missing coordinates of first pixels in the pixel array based on pixel information read from the plurality of second pixels, and generates an image based on the restored pixel information and the pixel information read from each first pixel.

10. A step of reading out pixel information at a first frame rate when light is incident on a pixel array, wherein the pixel array has a plurality of pixels arranged in a matrix, and the plurality of pixels include a plurality of first pixels whose pixel information is read out at a first frame rate when light is incident on the pixel array, and a plurality of second pixels whose pixel information is read out at a second frame rate higher than the first frame rate when light is incident on the pixel array, and the plurality of second pixels are arranged in the pixel array with spacing between them. Output an image generated from the pixel information read at the aforementioned first frame rate, or An imaging method comprising the steps of: reading out pixel information at the second frame rate when the light is incident on the pixel array; deriving an optical flow from the pixel information read out at the second frame rate; correcting an image generated from the pixel information read out at the first frame rate using the optical flow; and outputting the corrected image.

11. When outputting an image generated from pixel information read at the aforementioned first frame rate, The imaging method according to claim 10, comprising: restoring pixel information for missing coordinates of first pixels in the pixel array based on pixel information read from the plurality of second pixels; and generating the image based on the restored pixel information and the pixel information read at the first frame rate.

Images