Imaging device and imaging unit

Abstract

An imaging device comprising: a solid-state imaging element having a pixel array in which a plurality of pixels are arranged; and an imaging device main body that holds the solid-state imaging element, wherein the plurality of pixels include first pixels whose pixel information is read out at a first frame rate and second pixels whose pixel information is read out at a second frame rate that is higher than the first frame rate; and wherein the imaging device main body outputs an image generated from the pixel information read out at the first frame rate, or a corrected image obtained by correcting the image using an optical flow derived from the pixel information read out at the second frame rate.

Description

[Technical Field]

[0001] The present invention relates to an imaging device and an imaging unit. [Background technology]

[0002] Conventionally, imaging devices (such as smartphones) capable of capturing images of subjects such as people, objects, landscapes, etc. have been known. These imaging devices include an imaging unit having an optical system such as a lens and a solid-state imaging element such as an image sensor (Japanese Patent Application Laid-Open No. 2023-1788).

[0003] When capturing an image of an object using this imaging device, the captured image may be blurred or blurred if the object moves quickly or if the imaging device shakes during capture. This blurring or blurring in the captured image becomes particularly noticeable when the frame rate is reduced (i.e., the exposure time is extended) due to dim lighting or other reasons. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2023-1788 Summary of the Invention [Problem to be solved by the invention]

[0005] SUMMARY OF THE INVENTION It is therefore an object of the present invention to provide an imaging device and an imaging unit that can capture a clear image of a moving subject. [Means for solving the problem]

[0006] The imaging device according to the present invention comprises: a solid-state imaging device having a pixel array in which a plurality of pixels are arranged in a matrix; an imaging device body having an optical system, the solid-state imaging element is attached to the imaging device body so that light that has passed through the optical system is incident on the pixel array; The plurality of pixels are a plurality of first pixels from which pixel information is read out at a first frame rate when the light is incident on the pixel array; a plurality of second pixels from which pixel information when the light is incident on the pixel array is read out at a second frame rate higher than the first frame rate; The imaging device body outputs an image generated from pixel information read out at the first frame rate, or a corrected image obtained by correcting an image generated from pixel information read out at the first frame rate using an optical flow derived from pixel information read out at the second frame rate.

[0007] In the imaging device, The frame rate at which the pixel information is read out from the second pixels may be switchable between the second frame rate and the first frame rate.

[0008] In addition, in the imaging device, The second pixels may be spaced apart from one another in the pixel array.

[0009] In addition, in the imaging device, The first pixels may be greater than the second pixels.

[0010] In addition, in the imaging device, a first A / D converter to which the pixel information read from the first pixel is input; a second A / D converter to which the pixel information read from the second pixel is input, The resolution of the second A / D converter may be smaller than the resolution of the first A / D converter.

[0011] In addition, in the imaging device, the imaging element has a color filter in which red, green, and blue are arranged in a predetermined pattern and which is overlaid on the pixel array; The second pixel may be disposed at a position in the pixel array corresponding to a position where red or blue in the pattern of the color filter is to be disposed.

[0012] The portion of the color filter corresponding to the second pixel may be colorless and transparent.

[0013] The imaging unit according to the present invention further comprises: a solid-state imaging device having a pixel array in which a plurality of pixels are arranged in a matrix; an image generation unit that generates an image based on pixel information from the plurality of pixels; The plurality of pixels are a plurality of first pixels whose pixel information is read out at a first frame rate when light is incident on the pixel array; a plurality of second pixels from which pixel information when the light is incident on the pixel array is read out at a second frame rate higher than the first frame rate; The image generation unit generates and outputs an image from pixel information read at the first frame rate, or corrects an image generated from pixel information read at the first frame rate using an optical flow derived from pixel information read at the second frame rate and outputs the corrected image.

[0014] In the imaging unit, The image generation unit may restore pixel information of missing coordinates of first pixels in the pixel array based on pixel information read out from the plurality of second pixels, and generate an image based on the restored pixel information and the pixel information read out from each first pixel.

[0015] Further, the imaging method according to the present invention comprises: a step of reading out pixel information at a first frame rate when light is incident on a pixel array, wherein the pixel array includes a plurality of pixels arranged in a matrix, the plurality of pixels including a plurality of first pixels from which pixel information is read out at the first frame rate when light is incident on the pixel array, and a plurality of second pixels from which pixel information is read out at a second frame rate higher than the first frame rate when the light is incident on the pixel array; outputting an image generated from the pixel information read out in the first frame; or reading out pixel information when the light is incident on the pixel array at the second frame rate, deriving an optical flow from the pixel information read out at the second frame rate, correcting an image generated from the pixel information read out in the first frame using the optical flow, and outputting the corrected image.

[0016] The method comprises: When an image generated from pixel information read out at the first frame rate is output, Pixel information of missing coordinates of a first pixel in the pixel array may be restored based on pixel information read from the plurality of second pixels, and the image may be generated based on the restored pixel information and the pixel information read in the first frame. [Brief explanation of the drawings]

[0017] [Figure 1] FIG. 1 is a functional block diagram of a smartphone according to this embodiment. [Figure 2] FIG. 2 is a view of the smartphone as seen from the side where the optical system is arranged. [Figure 3] FIG. 3 is a diagram illustrating the configuration of a solid-state imaging element included in the smartphone. [Figure 4] FIG. 4 is a diagram illustrating the configuration of a color filter and a pixel array in the solid-state imaging device. DETAILED DESCRIPTION OF THE INVENTION

[0018] The imaging device according to this embodiment includes: a solid-state imaging device having a pixel array in which a plurality of pixels are arranged in a matrix; an imaging device body having an optical system, the solid-state imaging element is attached to the imaging device body so that light that has passed through the optical system is incident on the pixel array; The plurality of pixels are a plurality of first pixels from which pixel information is read out at a first frame rate when the light is incident on the pixel array; a plurality of second pixels from which pixel information when the light is incident on the pixel array is read out at a second frame rate higher than the first frame rate; The imaging device body outputs an image generated from pixel information read out at the first frame rate, or a corrected image obtained by correcting an image generated from pixel information read out at the first frame rate using an optical flow derived from pixel information read out at the second frame rate.

[0019] According to this configuration, an image of the object to be imaged is generated using pixel information from a first pixel among the multiple pixels possessed by one solid-state imaging element, and optical flow is derived using pixel information from the second pixel. As a result, even if the image generated using the pixel information from the first pixel is blurred or blurred due to the movement of the object to be imaged or the shaking of the imaging device, the image is corrected using the optical flow, thereby removing or suppressing the blurred or blurred image, and as a result, a clear image (corrected image) is output (displayed, etc.).

[0020] Furthermore, when the first pixel is continuously capturing images at a certain frame rate for capturing a moving image, the optical flow can be used to generate moving image information at a rate exceeding the original frame rate of the first pixel.

[0021] In addition, in the imaging device of this embodiment, The frame rate at which the pixel information is read out from the second pixels may be switchable between the second frame rate and the first frame rate.

[0022] According to this configuration, an image can be generated using not only pixel information from the first pixel but also pixel information read out from the second pixel at the first frame rate, thereby further improving the image quality of the generated image.

[0023] In addition, in the imaging device of this embodiment, The second pixels may be spaced apart from one another in the pixel array.

[0024] By arranging the second pixels at intervals in this manner, pixel information for deriving optical flow can be obtained from the entire pixel array, thereby further improving the quality of the corrected image.

[0025] In addition, in the imaging device of this embodiment, The first pixels may be greater than the second pixels.

[0026] With this configuration, the number of pixels required to generate an image is ensured, and even if second pixels (pixels for deriving optical flow) are arranged in the pixel array, degradation in the image quality of the generated image is suppressed.

[0027] In addition, the imaging device of this embodiment a first A / D converter to which the pixel information read from the first pixel is input; a second A / D converter to which the pixel information read from the second pixel is input, The resolution of the second A / D converter may be smaller than the resolution of the first A / D converter.

[0028] By reducing the resolution in this way, it is easy to ensure the processing speed of the second A / D converter, and as a result, pixel information read at the second frame rate (high frame rate) can be suitably processed.

[0029] In addition, in the imaging device of this embodiment, the solid-state imaging device has a color filter in which red, green, and blue are arranged in a predetermined pattern and which is overlaid on the pixel array; The second pixel may be disposed at a position in the pixel array corresponding to a position where red or blue in the pattern of the color filter is to be disposed.

[0030] Since humans are sensitive to changes in green brightness (resolution), by ensuring the number of first pixels of the green filter as in the above configuration, even if the number of pixels (first pixels) used to generate the image is reduced by arranging second pixels in the pixel array, degradation in the image quality of the generated image is suppressed.

[0031] in this case, The portion of the color filter corresponding to the second pixel may be colorless and transparent.

[0032] According to this configuration, it is possible to suppress a decrease in the amount of light received by the second pixel caused by the color filter.

[0033] Moreover, the imaging unit according to this embodiment has: a solid-state imaging device having a pixel array in which a plurality of pixels are arranged in a matrix; an image generation unit that generates an image based on pixel information from the plurality of pixels; The plurality of pixels are a plurality of first pixels whose pixel information is read out at a first frame rate when light is incident on the pixel array; a plurality of second pixels from which pixel information when the light is incident on the pixel array is read out at a second frame rate higher than the first frame rate; The image generation unit generates and outputs an image from pixel information read at the first frame rate, or corrects an image generated from pixel information read at the first frame rate using an optical flow derived from pixel information read at the second frame rate and outputs the corrected image (corrected image).

[0034] According to this configuration, an image of the object to be imaged is generated using pixel information from a first pixel among the multiple pixels possessed by one solid-state imaging element, and optical flow is derived using pixel information from the second pixel. As a result, even if the image generated using the pixel information from the first pixel is blurred or blurred due to the movement of the object to be imaged or the shaking of the imaging device, the image is corrected using the optical flow, thereby removing or suppressing the blurred or blurred image, and as a result, a clear image (corrected image) is output.

[0035] Furthermore, when the first pixel is capturing video continuously at a certain frame rate, optical flow can be used to generate and output video information that exceeds the original frame rate of the first pixel.

[0036] In addition, in the imaging unit of this embodiment, The image generation unit may restore pixel information of missing coordinates of first pixels in the pixel array based on pixel information read out from the plurality of second pixels, and generate an image based on the restored pixel information and the pixel information read out from each first pixel.

[0037] In this way, at the position where the second pixel is placed in the pixel array, the pixel information of the missing coordinates of the first pixel, i.e., the pixel information that would be obtained (read) from the first pixel if the first pixel were placed at the position of the second pixel, is restored (predicted) and used to generate the image, thereby further improving the image quality of the generated image.

[0038] In addition, the imaging method according to this embodiment includes: a step of reading out pixel information at a first frame rate when light is incident on a pixel array, wherein the pixel array includes a plurality of pixels arranged in a matrix, the plurality of pixels including a plurality of first pixels from which pixel information is read out at the first frame rate when light is incident on the pixel array, and a plurality of second pixels from which pixel information is read out at a second frame rate higher than the first frame rate when the light is incident on the pixel array; outputting an image generated from the pixel information read out in the first frame; or The method includes steps of reading out pixel information when the light is incident on the pixel array at the second frame rate, deriving an optical flow from the pixel information read out at the second frame rate, correcting an image generated from the pixel information read out in the first frame using the optical flow, and outputting the corrected image (corrected image).

[0039] According to this method, an image of the object to be imaged is generated using pixel information from a first pixel among a plurality of pixels, and optical flow is derived using pixel information from a second pixel. As a result, even if the image generated using the pixel information from the first pixel is blurred or blurred due to the movement of the object to be imaged or the shaking of the imaging device, the image is corrected using the optical flow, thereby removing or suppressing the blurred or blurred image, and as a result, a clear image (corrected image) is output.

[0040] Furthermore, when the first pixel is capturing video continuously at a certain frame rate, optical flow can be used to generate and output video information that exceeds the original frame rate of the first pixel.

[0041] In addition, the method of this embodiment is When an image generated from pixel information read out at the first frame rate is output, Pixel information of missing coordinates of a first pixel in the pixel array may be restored based on pixel information read from the plurality of second pixels, and the image may be generated based on the restored pixel information and the pixel information read in the first frame.

[0042] In this way, at the position where the second pixel is placed in the pixel array, the pixel information of the missing coordinates of the first pixel, i.e., the pixel information that would be obtained (read) from the first pixel if the first pixel were placed at the position of the second pixel, is restored (predicted) and used to generate the image, thereby further improving the image quality of the generated image.

[0043] As described above, according to this embodiment, it is possible to provide an imaging device and an imaging unit that can clearly capture an image of a moving subject.

[0044] Hereinafter, an embodiment of the present invention will be described with reference to the accompanying drawings.

[0045] The imaging device is capable of capturing still images and moving images, and is a digital camera, a smartphone, a tablet device, etc. The imaging device 1 of this embodiment is a smartphone, as shown in FIGS.

[0046] 3 , the smartphone (imaging device) 1 includes a solid-state imaging element 2 having a pixel array 21 and a smartphone body (imaging device body) 3 having an optical system 31 configured with at least one optical element such as a lens, and the solid-state imaging element 2 is attached to the smartphone body 3 so that light passing through the optical system 31 is incident on the pixel array 21. The smartphone 1 also includes a control unit 4 connected to the solid-state imaging element 2. The smartphone 1 of this embodiment also includes a non-volatile memory 101, a working memory 102, an operation unit 103, a display unit 104, a recording medium 105, a connection unit 106, a short-range wireless communication unit 107, a public network connection unit 108, a microphone 109, and a speaker 110.

[0047] Under the control of the control unit 4, this smartphone 1 converts the image of the object (subject) formed by the optical system 31 of the smartphone main body 3 into an electrical signal, then performs noise reduction processing, etc., and outputs the digital data as an output image (image data) (for example, display on the display unit 104, etc.).

[0048] The solid-state imaging element 2 is an element for capturing an image of an object to be imaged, more specifically, an element for converting an image of the object to be imaged formed through the optical system 31 into an electrical signal, and the solid-state imaging element 2 of this embodiment is, for example, a CMOS image sensor.

[0049] As also shown in Figure 4, this solid-state imaging element 2 has a pixel array 21, a color filter 25 overlaid on the pixel array 21, and an A / D conversion unit 26 that A / D converts signals (pixel information) read from each pixel 22.

[0050] The color filter 25 limits the wavelength of light incident on each pixel 22, and filters that pass red, green, and blue wavelengths are arranged in a predetermined pattern. The arrangement pattern of the red, green, and blue filters in the color filter 25 of this embodiment is a Bayer array. Note that the arrangement pattern is not limited to the red, green, and blue Bayer array, and other colors and arrangement patterns may be used.

[0051] In the pixel array 21, a plurality of pixels 22, each having a photoelectric conversion element FD, are arranged in a matrix (see FIG. 3). That is, the pixel array 21 is made up of a plurality of pixels 22 arranged in a matrix. Note that FIG. 3 shows only a part of the pixel array 21, and in the pixel array 21, the arrangement shown in FIG. 3 is repeated in the row direction (left-right direction in FIG. 3) and column direction (up-down direction in FIG. 3). Furthermore, in each pixel 22 in FIGS. 3 and 4, a pattern such as a dot is added to indicate the difference in color of the color filter 25 (see FIG. 4).

[0052] These multiple pixels 22 include multiple first pixels 23 from which pixel information (voltage signals corresponding to signal charges generated by the photoelectric conversion elements FD of each pixel 22) is read out at a first frame rate when light that has passed through the optical system 31 is incident on the pixel array 21, and multiple second pixels 24 from which pixel information is read out at a second frame rate that is higher than the first frame rate when light that has passed through the optical system 31 is incident on the pixel array 21. The first pixels 23 and the second pixels 24 in this embodiment have the same configuration, and only the frame rate at which the pixel information is read out is different.

[0053] In this embodiment, the first frame rate is, for example, 60 fps, and the second frame rate is, for example, 1000 fps. The first frame rate may be the readout rate of the RGB pixels of the CMOS image sensor, measured in units of times per second. The second frame rate may be the readout rate of the EVS pixels of the CMOS image sensor, measured in units of times per second. Typically, the second frame rate is much higher than the first frame rate. Note that the frame rates at which pixel information is read from each pixel 23, 24 are not limited to these values.

[0054] In addition, in the pixel array 21 of this embodiment, the number of first pixels 23 is greater than the number of second pixels 24. For example, the ratio of the number of first pixels 23 to the number of second pixels 24 in the pixel array 21 is 8:1, 16:1, 32:1, or the like.

[0055] The second pixels 24 are arranged at intervals in the pixel array 21 (see FIG. 4). Specifically, the second pixels 24 are arranged approximately evenly across the pixel array 21 so that the intervals between adjacent second pixels 24 are approximately the same.

[0056] Furthermore, each second pixel 24 is arranged in a position in the pixel array 21 that corresponds to a planned position for arranging red or blue in the arrangement pattern (Bayer array in the example of this embodiment) of the color filter 25. The portion of the color filter 25 that corresponds to the second pixel 24 is colorless and transparent. That is, in the color filter 25 of this embodiment, the number of red or blue filters is reduced without reducing the number of green filters in the arrangement in the Bayer array, and the position where the red or blue filters have been reduced is made colorless and transparent.

[0057] The A / D conversion unit 26 converts pixel information (analog signals) from the pixel array 21 into digital signals and outputs them to a subsequent stage. The A / D conversion unit 26 of this embodiment has a plurality of first A / D converters 261 to which pixel information read out from the first pixels 23 is input, and a plurality of second A / D converters 262 to which pixel information read out from the second pixels 24 is input. These first A / D converters 261 and second A / D converters 262 are respectively arranged for each column (column of pixels 22) of the pixel array 21 in which the pixels 22 are arranged in a matrix.

[0058] Each first A / D converter 261 is connected to each first pixel 23 in the column direction in the pixel array 21 via a first signal line 27a, and each second A / D converter 262 is connected to each second pixel 24 in the column direction in the pixel array 21 via a second signal line 27b. In the A / D conversion unit 26 of this embodiment, the resolution of the second A / D converter 262 is smaller than the resolution of the first A / D converter 261. For example, the resolution of the second A / D converter 262 of this embodiment is 6 bits, and the resolution of the first A / D converter 261 of this embodiment is 10 bits. Note that in FIG. 3 , the line connected to each first pixel 23 in the row direction is the first control line 28a, and the line connected to each second pixel 24 in the row direction is the second control line 28b. These control lines 28a and 28b are wiring for controlling each pixel 23 and 24, such as resetting each pixel 23 and 24 and selecting a row from which pixel information (signals) is read out.

[0059] The control unit 4 controls each unit of the smartphone 1 in accordance with input signals and programs. The control unit 4 also generates a captured image (an image for external output) from a signal (pixel information) output from the solid-state imaging device 2, and outputs the captured image to the display unit 104. Note that the control unit that controls each unit of the smartphone 1 and the control unit 4 that performs control related to imaging may be provided separately.

[0060] The control unit 4 has an arithmetic processing unit (image generation unit) 41 connected directly or indirectly to the solid-state imaging element 2, and the arithmetic processing unit 41 derives an optical flow from pixel information read from the second pixels 24, generates an image from pixel information read from the first pixels 23, and corrects the generated image using the derived optical flow. The arithmetic processing unit 41 of this embodiment functionally has an image generation unit that generates an image from pixel information read from the first pixels 23, an optical flow calculation unit that derives an optical flow from pixel information read from the second pixels 24, and a correction calculation unit that corrects the image using the optical flow.

[0061] Here, optical flow refers to the movement of each optical point between two temporally consecutive image frames, which can be observed on a digital image by projecting the movement of an object to be imaged onto the image, and is, for example, a vectorized difference between feature points (moving points) between two temporally adjacent image frames in a video. In the smartphone 1 of this embodiment, in the pixel array 21 of the solid-state imaging device 2, multiple images are captured by increasing the frame rate (second frame rate) when reading pixel information from the second pixels 24 higher than the frame rate (first frame rate) when reading pixel information from the first pixels 23, and the optical flow is derived by the arithmetic processing unit 41 calculating the vectors (moving direction and distance) of each moving point from each of two temporally adjacent image pairs in the multiple images acquired by this capturing (images captured by the second pixels 24).

[0062] Furthermore, when deriving (calculating) the optical flow from pixel information read from the second pixels 24, the arithmetic processing unit 41 creates an event signal quantized into three values ​​(+1 / 0 / −1) from the difference between consecutive images (images based on pixel information read from each second pixel 24), reduces the data transmission bandwidth, and then derives the optical flow using the created event signal. Here, +1 indicates an increase in luminance by a certain percentage or more, 0 indicates an increase in luminance that is within a certain range, and −1 indicates an increase in luminance that is below a certain level. Furthermore, when the arithmetic processing unit 41 creates an event signal from pixel information read from the second pixels 24, the event signal may be created based on a luminance signal (Y). If there is sufficient data transmission bandwidth, the arithmetic processing unit 41 may derive the optical flow by directly calculating the luminance signal with gradation. Furthermore, the method of deriving (calculating) the optical flow is not limited to the above-described method. The optical flow may be derived using various conventional methods (for example, the gradient method, the Lucas-Kanade method, etc.).

[0063] The nonvolatile memory 101 is an electrically erasable and recordable nonvolatile memory. In this embodiment, the nonvolatile memory 101 stores an OS (operating system), which is basic software executed by the control unit 4, and applications that cooperate with the OS to realize applied functions.

[0064] The work memory 102 is used as an image display memory for the display unit 104, a work area for the control unit 4, and the like.

[0065] The operation unit 103 is used by a user or the like to input instructions to the smartphone 1. The operation unit 103 of this embodiment has a power button for instructing the smartphone 1 to be powered on / off, a touch panel formed on the display unit 104, and the like.

[0066] The display unit 104 displays (outputs to the outside) captured images (image data) and displays characters for operation.

[0067] The recording medium 105 records the image data (captured image) output from the control unit 4.

[0068] The connection unit 106 is an interface for connecting to an external device. The smartphone 1 exchanges data with the external device via the connection unit 106.

[0069] The short-distance wireless communication unit 107 is a communication unit for performing short-distance wireless communication, and is configured with an antenna for wireless communication, a modulation / demodulation circuit for processing wireless signals, and a communication controller.

[0070] The public network connection unit 108 is an interface for performing public wireless communication. The smartphone 1 communicates with other devices via this public network connection unit 108 to make a call. At this time, the control unit 4 realizes the call by inputting and outputting audio signals via the microphone 109 and the speaker 110. The public network connection unit 108 in this embodiment is an antenna, and the control unit 4 connects to the public network via this antenna.

[0071] The smartphone 1 configured as above can correct shaking and blurring of a captured image in the following manner.

[0072] An image of an object is captured by the smartphone 1. At this time, in the solid-state imaging element 2, pixel information (first image) is read out from each first pixel 23 at 60 frames per second (fps), while pixel information (second image) is read out from each second pixel 24 at 1000 fps. Specifically, in FIG. 3 , the plurality of first pixels 23 corresponding to the first control line 28a are scanned sequentially from top to bottom 60 times per second, while the plurality of second pixels 24 corresponding to the second control line 28b are scanned sequentially from top to bottom 1000 times per second, independently of this.

[0073] In addition, when not all of the second A / D converters 262 arranged in the row direction in Figure 3 are occupied by pixel information, multiple second control lines 28b may be activated simultaneously, and pixel information may be input to the multiple second A / D converters 262 from the second pixels 24 connected via each second signal line 27b.

[0074] When a plurality of second images (images based on pixel information read from each second pixel 24) are read from each second pixel 24 in the pixel array 21, the arithmetic processing unit 41 derives an optical flow and corrects the corresponding first image (images based on pixel information read from each first pixel 23) using this derived optical flow. This corrects shaking and blurring in the first image due to the movement of the imaging subject, etc., and a clear image (corrected image) is obtained.

[0075] In addition, in this corrected image, the control unit 4 of this embodiment determines the reliability of the corrected captured image using optical flow, and if the reliability is low, outputs the captured image without correction (an image generated using pixel information read out from the first pixel 23).

[0076] The reliability is determined, for example, by object recognition using AI, etc. In this case, for example, when a human face is not determined to be a human face, the reliability is determined to be low.

[0077] Furthermore, a second image based on pixel information read from the second pixel 24 may be used to determine the reliability. Specifically, the second image output from the second pixel 24 has a low resolution and a poor S / N ratio, but contains unblurred information, and therefore the reliability of the correction result is determined based on this second image. Note that the reliability of the corrected first image may be determined using optical flow, and if the reliability is low, correction may be performed again using the second image output from the second pixel 24. Specifically, the second image output from the second pixel 24 has a low resolution and a poor S / N ratio, but contains unblurred information, and therefore the image information from the first pixel 23 is replaced or blended based on this image information.

[0078] The above-described smartphone (imaging device) 1 includes a solid-state imaging element 2 having a pixel array 21 in which a plurality of pixels 22 are arranged in a matrix, and a smartphone body (imaging device body) 3 having an optical system 31. The solid-state imaging element 2 is attached to the smartphone body 3 so that light passing through the optical system 31 is incident on the pixel array 21. The plurality of pixels 22 include a plurality of first pixels 23 from which pixel information when light is incident on the pixel array 21 is read out at a first frame rate, and a plurality of second pixels 24 from which pixel information when light is incident on the pixel array 21 is read out at a second frame rate higher than the first frame rate. The smartphone body 3 outputs an image generated from the pixel information read out at the first frame rate, or a corrected image obtained by correcting an image generated from the pixel information read out at the first frame rate using an optical flow derived from pixel information read out at the second frame rate.

[0079] According to this configuration, an image of the object to be imaged is generated using pixel information from the first pixel 23 among the plurality of pixels 22 that one solid-state imaging element 2 has, and optical flow is derived using pixel information from the second pixel 24. As a result, even if the image generated using the pixel information from the first pixel 23 is blurred or blurred due to the movement of the object to be imaged or the shaking of the imaging device, the image is corrected using the optical flow, thereby removing or suppressing the blurred or blurred image, and as a result, a clear image (corrected image) is output.

[0080] Furthermore, when the first pixel 23 is continuously capturing images at a certain frame rate for capturing a moving image, it is possible to generate moving image information at a rate exceeding the original frame rate of the first pixel 23 by using the optical flow.

[0081] In the smartphone 1 of this embodiment, the second pixels 24 are arranged in the pixel array 21 at intervals.

[0082] By arranging the second pixels 24 at intervals from one another in this manner, pixel information for deriving optical flow can be obtained from the entire area of ​​the pixel array 21, and the quality of the corrected image is further improved.

[0083] Furthermore, in the smartphone 1 of this embodiment, the number of first pixels 23 is greater than the number of second pixels 24. Therefore, the number of pixels (the number of first pixels 23) for generating an image is ensured, and even if second pixels (pixels for deriving optical flow) 24 are arranged in the pixel array 21, degradation in the quality of the generated image is prevented.

[0084] In addition, the smartphone 1 of this embodiment is equipped with a first A / D converter to which pixel information read out from the first pixel 23 is input, and a second A / D converter to which pixel information read out from the second pixel 24 is input, and the resolution of the second A / D converter is smaller than the resolution of the first A / D converter.

[0085] According to this configuration, it is easy to ensure the processing speed of the second A / D converter, and as a result, pixel information read at the second frame rate is suitably processed.

[0086] In addition, in the smartphone 1 of this embodiment, the solid-state imaging element 2 has a color filter 25 in which red, green, and blue are arranged in a predetermined pattern and which is overlaid on the pixel array 21, and the second pixel 24 is arranged in a position in the pixel array 21 corresponding to the intended position of red or blue in the pattern of the color filter 25.

[0087] Since humans are sensitive to changes in green brightness (resolution), by ensuring the number of first pixels 23 with green filters superimposed thereon as in the above configuration, even if the number of pixels (first pixels) 23 used to generate an image is reduced by arranging second pixels 24 in the pixel array 21, the degradation of the generated image quality is suppressed.

[0088] In the smartphone 1 of this embodiment, the portions of the color filter 25 corresponding to the second pixels 24 are colorless and transparent.

[0089] According to this configuration, a decrease in the amount of light received by the second pixel 24 due to the color filter 25 can be suppressed.

[0090] The imaging device of the present invention is not limited to the above-described embodiment, and various modifications can be made without departing from the spirit of the present invention. For example, the configuration of one embodiment can be added to the configuration of another embodiment, or part of the configuration of one embodiment can be replaced with the configuration of another embodiment. Furthermore, part of the configuration of one embodiment can be deleted.

[0091] The imaging device 1 in the above embodiment is a smartphone, but is not limited to this configuration. The imaging device 1 may be another device (device having an imaging function) such as a digital camera or a tablet device (mobile terminal).

[0092] Furthermore, although the imaging device 1 of the above embodiment includes one imaging unit (optical system 31 and solid-state imaging element 2 corresponding to the optical system 31), this configuration is not limited thereto. The imaging device 1 may include multiple imaging units. For example, the imaging device 1 may include a telephoto imaging unit capable of telephoto imaging, a wide-angle imaging unit capable of wide-angle imaging, and an ultra-wide-angle imaging unit capable of ultra-wide-angle imaging. In this case, the focal length of the telephoto imaging unit is 80 mm or more in 35 mm film equivalent, the focal length of the wide-angle imaging unit is 24 mm in 35 mm film equivalent, and the focal length of the ultra-wide-angle imaging unit is 14 mm or less in 35 mm film equivalent.

[0093] In this case, the pixel array 21 of the solid-state imaging element 2 of each imaging unit may have first pixels 23 and second pixels 24, and in each imaging unit, imaging may be performed at a first frame rate by the first pixels 23 and at a second frame rate by the second pixels 24. In other words, the configuration may be such that image generation and optical flow derivation are performed for each imaging unit.

[0094] Furthermore, for example, only the solid-state imaging element 2 of the imaging element with the widest angle of view among the multiple imaging elements (in the above example, the ultra-wide-angle imaging element) may have the first pixel 23 and the second pixel 24, and the solid-state imaging elements 2 of the remaining imaging elements (in the above example, the wide-angle imaging element and the telephoto imaging element) may have only the first pixel 23. In this case, regardless of which imaging element is used to capture an image, an optical flow is derived by imaging at the second pixel 24 of the ultra-wide-angle imaging element, and the image captured at the first pixel 23 of each imaging element is corrected by this optical flow.

[0095] Furthermore, the imaging device 1 of the above embodiment derives an optical flow for each imaging session and determines whether to correct the image (the image generated from pixel information from the first pixel 23) using the derived optical flow based on the reliability, but this configuration is not limited to this. A configuration in which the photographer manually selects whether to derive the optical flow is also possible. Furthermore, for example, a configuration in which the optical flow is automatically switched to be derived when the accumulation time of incident light in the solid-state imaging element 2 for normal imaging is 1 / 1000 seconds or longer may be used.

[0096] Furthermore, in the imaging device 1 of the above embodiment, pixel information is read from the second pixels 24 at the second frame rate, but this configuration is not limited to this. The imaging device 1 may be configured to be able to switch the frame rate at which pixel information is read from the second pixels 24 between the second frame rate and the first frame rate. If the image generated using the pixel information from the first pixels 23 does not suffer from blurring or blurring, the image quality of the generated image can be further improved by reading the pixel information from the second pixels 24 at the first frame rate and using it to generate the image.

[0097] Furthermore, the arithmetic processing unit 41 of the imaging device 1 in the above embodiment is included in the control unit 4 that controls the imaging device (smartphone) 1, but is not limited to this configuration. The solid-state imaging element 2 may have the arithmetic processing unit 41. Alternatively, the solid-state imaging element 2 and the arithmetic processing unit (image generating unit) 41 may form an imaging unit, and this imaging unit may be mounted on the imaging device 1.

[0098] In this imaging unit, the image generation unit may restore (predict) pixel information at missing coordinates of a first pixel 23 in the pixel array 21, based on pixel information read from a plurality of second pixels 24, i.e., pixel information that would be obtained (read) from the first pixel 23 if the first pixel 23 were located at the position of the second pixel 24, and generate an image based on the restored pixel information and the pixel information read from each first pixel 23. In this case, when generating an image by the image generation unit, the restoration may be performed together with correction based on optical flow, or the restoration may be performed with optical flow correction turned off. Note that the restoration (reconstruction (calculation, prediction, etc.) of pixel information at the missing coordinates) may be calculated by calculation or the like based on pixel information read from first pixels 23 surrounding the coordinates, or may be predicted or the like using AI or the like.

[0099] In this way, at the position where the second pixel 24 is located in the pixel array 21, the pixel information of the missing coordinates of the first pixel 23, i.e., the pixel information that would be obtained (read out) from the first pixel 23 if the first pixel 23 were located at the position where the second pixel 24 is located, is restored (predicted, etc.) and used to generate the image, thereby further improving the image quality of the generated image.

[0100] Furthermore, in the imaging device 1 of the above embodiment, the first pixel 23 and the second pixel 24 of the solid-state imaging element 2 have the same configuration and operation, but this configuration is not limiting. The first pixel 23 and the second pixel 24 may have different configurations and operations. Since the second pixel 24 is required to output a large signal in a short exposure time, the second pixel 24 may be designed or set, for example, to have a conversion gain for converting signal charge into voltage that is larger than the first gain, which is the conversion gain of the first pixel 23, but this configuration is not limiting.

[0101] In order to express the present invention, the present invention has been properly and sufficiently described above through the embodiments with reference to the drawings, but it should be recognized that those skilled in the art can easily change and / or improve the above-mentioned embodiments. Therefore, unless the changes or improvements made by those skilled in the art are at a level that causes departure from the scope of the claims described in the claims, such changes or improvements are interpreted as being included in the scope of the claims. [Explanation of symbols]

[0102] 1...smartphone (imaging device), 2...solid-state imaging element, 21...pixel array, 22...pixel, 23...first pixel, 24...second pixel, 25...color filter, 26...A / D conversion unit, 261...first A / D converter, 262...second A / D converter, 27a...first signal line, 27b...second signal line, 28a...first control line, 28b...second control line, 3...smartphone body, 31...optical system, 4...control unit, 41...arithmetic processing unit, 101...non-volatile memory, 102...working memory, 103...operation unit, 104...display unit, 105...recording medium, 106...connection unit, 107...near-field wireless communication unit, 108...public network connection unit, 109...microphone, 110...speaker, FD...photoelectric conversion element

Claims

1. a solid-state imaging device having a pixel array in which a plurality of pixels are arranged in a matrix; an imaging device body having an optical system, the solid-state imaging element is attached to the imaging device body so that light that has passed through the optical system is incident on the pixel array; The plurality of pixels are a plurality of first pixels from which pixel information is read out at a first frame rate when the light is incident on the pixel array; a plurality of second pixels from which pixel information when the light is incident on the pixel array is read out at a second frame rate higher than the first frame rate; The imaging device main body outputs an image generated from pixel information read at the first frame rate, or a corrected image obtained by correcting an image generated from pixel information read at the first frame rate using an optical flow derived from pixel information read at the second frame rate.

2. The imaging device according to claim 1 , wherein a frame rate at which the pixel information is read from the second pixels can be switched between the second frame rate and the first frame rate.

3. The imaging device according to claim 1 , wherein the second pixels are arranged at intervals in the pixel array.

4. 4. The imaging device according to claim 1, wherein the number of the first pixels is greater than the number of the second pixels.

5. a first A / D converter to which the pixel information read from the first pixel is input; a second A / D converter to which the pixel information read from the second pixel is input, 5. The imaging device according to claim 1, wherein the resolution of the second A / D converter is smaller than the resolution of the first A / D converter.

6. the solid-state imaging device has a color filter in which red, green, and blue are arranged in a predetermined pattern and which is overlaid on the pixel array; 6. The imaging device according to claim 1, wherein the second pixel is disposed in a position in the pixel array corresponding to a position where red or blue is to be disposed in the pattern of the color filter.

7. The imaging device according to claim 6 , wherein a portion of the color filter corresponding to the second pixel is colorless and transparent.

8. a solid-state imaging device having a pixel array in which a plurality of pixels are arranged in a matrix; an image generation unit that generates an image based on pixel information from the plurality of pixels; The plurality of pixels are a plurality of first pixels whose pixel information is read out at a first frame rate when light is incident on the pixel array; a plurality of second pixels from which pixel information when the light is incident on the pixel array is read out at a second frame rate higher than the first frame rate; The image generation unit generates and outputs an image from pixel information read out at the first frame rate, or corrects an image generated from pixel information read out at the first frame rate using an optical flow derived from pixel information read out at the second frame rate and outputs the corrected image.

9. 9. The imaging unit according to claim 8, wherein the image generation unit restores pixel information of missing coordinates of first pixels in the pixel array based on pixel information read out from the plurality of second pixels, and generates an image based on the restored pixel information and the pixel information read out from each first pixel.

10. a step of reading out pixel information at a first frame rate when light is incident on a pixel array, wherein the pixel array includes a plurality of pixels arranged in a matrix, the plurality of pixels including a plurality of first pixels from which pixel information is read out at the first frame rate when light is incident on the pixel array, and a plurality of second pixels from which pixel information is read out at a second frame rate higher than the first frame rate when the light is incident on the pixel array; outputting an image generated from the pixel information read out in the first frame; or reading out pixel information at the second frame rate when the light is incident on the pixel array, deriving an optical flow from the pixel information read out at the second frame rate, correcting an image generated from the pixel information read out in the first frame using the optical flow, and outputting the corrected image. Imaging method.

11. When an image generated from pixel information read out at the first frame rate is output, restoring pixel information of missing coordinates of the first pixel in the pixel array based on pixel information read out from the plurality of second pixels, and generating the image based on the restored pixel information and the pixel information read out in the first frame; The imaging method according to claim 10.

Images