Photographing apparatus and photographing unit

Abstract

A photographing apparatus, comprising: a solid photographing element, which is provided with a pixel array configured with a plurality of pixels; and a photographing apparatus main body, which holds the solid photographing element, wherein the plurality of pixels comprise a first pixel, which is used for reading out pixel information at a first frame rate, and a second pixel, which is used for reading out the pixel information at a second frame rate higher than the first frame rate, and the photographing apparatus main body outputs an image generated on the basis of the pixel information that is read out at the first frame rate, or outputs a corrected image obtained by means of correcting the image by using an optical flow exported from the pixel information that is read out at the second frame rate.

Description

Camera and camera unit Technical Field

[0001] The present invention relates to a photographing device and a photographing unit. Background Art

[0002] Conventionally, there are known imaging devices (smartphones, etc.) that can capture images of subjects such as people, objects, and scenery. These imaging devices include an imaging unit having an optical system such as a lens and a solid-state imaging element such as an image sensor (Japanese Patent Application Laid-Open No. 2023-1788).

[0003] When photographing a subject using such a camera, jitter and defocus may occur in the resulting image due to the subject's rapid motion or shaking of the camera during shooting. This jitter and defocus may become noticeable in the captured image, particularly when the frame rate is reduced (i.e., the exposure time is extended) due to, for example, low ambient light.

[0004] Prior art literature

[0005] Patent Literature

[0006] Patent Document 1: Japanese Patent Application Laid-Open No. 2023-1788

[0007] Summary of the Invention

[0008] Problems to be solved by the invention

[0009] Therefore, an object of the present invention is to provide an imaging device and an imaging unit capable of clearly capturing a moving subject.

[0010] Solutions for solving problems

[0011] The photographing device of the present invention comprises:

[0012] a solid-state imaging element having a pixel array in which a plurality of pixels are arranged in a matrix; and

[0013] A camera body having an optical system,

[0014] The solid-state imaging element is mounted on the imaging device body so that light passing through the optical system is incident on the pixel array.

[0015] The plurality of pixels comprises:

[0016] a plurality of first pixels configured to read out pixel information when the light is incident on the pixel array at a first frame rate; and

[0017] a plurality of second pixels configured to read out pixel information when the light is incident on the pixel array at a second frame rate higher than the first frame rate;

[0018] The shooting device body outputs an image generated based on the pixel information read out at the first frame rate, or outputs a corrected image, and the corrected image is obtained in the following manner: deriving an optical flow from the pixel information read out at the second frame rate, and using the optical flow to correct the image generated based on the pixel information read out at the first frame rate, thereby obtaining the corrected image.

[0019] In the shooting device,

[0020] The frame rate at which the pixel information is read out from the second pixel may be switched between the second frame rate and the first frame rate.

[0021] In addition, in the shooting device,

[0022] The plurality of second pixels may be arranged at intervals from each other in the pixel array.

[0023] In addition, in the shooting device,

[0024] It is also possible that the first pixels are more than the second pixels.

[0025] In addition, the photographing device may further include:

[0026] a first A / D converter to which the pixel information read out from the first pixel is input; and

[0027] a second A / D converter to which the pixel information read out from the second pixel is input,

[0028] A resolution of the second A / D converter is smaller than a resolution of the first A / D converter.

[0029] In addition, in the shooting device,

[0030] The imaging element may include a color filter in which red, green, and blue are arranged in a predetermined pattern and overlap with the pixel array.

[0031] The second pixel is arranged in the pixel array at a position corresponding to a planned arrangement position of red or blue in the pattern of the color filter.

[0032] The portion of the color filter corresponding to the second pixel may be colorless and transparent.

[0033] In addition, the imaging unit of the present invention comprises:

[0034] a solid-state imaging element having a pixel array in which a plurality of pixels are arranged in a matrix; and

[0035] an image generating unit that generates an image based on pixel information from the plurality of pixels,

[0036] The plurality of pixels comprises:

[0037] a plurality of first pixels configured to read out pixel information when the light is incident on the pixel array at a first frame rate; and

[0038] a plurality of second pixels configured to read out pixel information when the light is incident on the pixel array at a second frame rate higher than the first frame rate;

[0039] The image generation unit generates and outputs an image based on the pixel information read out at the first frame rate, or corrects the image generated based on the pixel information read out at the first frame rate using the optical flow derived from the pixel information read out at the second frame rate, and outputs the corrected image.

[0040] In the shooting unit,

[0041] The image generating unit may also restore pixel information of missing coordinates of the first pixel in the pixel array based on pixel information read from the plurality of second pixels, and generate an image based on the restored pixel information and pixel information read from each first pixel.

[0042] In addition, the photographing method of the present invention includes:

[0043] Reading out pixel information when light is incident on a pixel array at a first frame rate, wherein the pixel array is configured with a plurality of pixels in a matrix, the plurality of pixels comprising: a plurality of first pixels for reading out pixel information when light is incident on the pixel array at the first frame rate; and a plurality of second pixels for reading out pixel information when the light is incident on the pixel array at a second frame rate higher than the first frame rate; and

[0044] outputting an image generated based on the pixel information read out at the first frame rate, or

[0045] Pixel information when the light is incident on the pixel array is read out at the second frame rate, an optical flow is derived from the pixel information read out at the second frame rate, and an image generated based on the pixel information read out at the first frame rate is corrected using the optical flow, and the corrected image is output.

[0046] In the shooting method,

[0047] It is also possible to output an image generated based on the pixel information read out at the first frame rate,

[0048] Pixel information of missing coordinates of a first pixel in the pixel array is restored based on pixel information read from the plurality of second pixels, and the image is generated based on the restored pixel information and pixel information read at the first frame rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] FIG1 is a functional block diagram of a smartphone according to this embodiment.

[0050] FIG. 2 is a diagram showing the smartphone as viewed from the side where the optical system is provided.

[0051] FIG. 3 is a diagram for explaining the configuration of a solid-state imaging element included in the smartphone.

[0052] FIG. 4 is a diagram for explaining the configuration of a color filter and a pixel array in the solid-state imaging element. DETAILED DESCRIPTION

[0053] The imaging device of this embodiment includes:

[0054] a solid-state imaging element having a pixel array in which a plurality of pixels are arranged in a matrix; and

[0055] A camera body having an optical system,

[0056] The solid-state imaging element is mounted on the imaging device body so that light passing through the optical system is incident on the pixel array.

[0057] The plurality of pixels comprises:

[0058] a plurality of first pixels configured to read out pixel information when the light is incident on the pixel array at a first frame rate; and

[0059] a plurality of second pixels configured to read out pixel information when the light is incident on the pixel array at a second frame rate higher than the first frame rate;

[0060] The shooting device body outputs an image generated based on the pixel information read out at the first frame rate, or outputs a corrected image, and the corrected image is obtained in the following manner: deriving an optical flow from the pixel information read out at the second frame rate, and using the optical flow to correct the image generated based on the pixel information read out at the first frame rate, thereby obtaining the corrected image.

[0061] According to the above-mentioned configuration, an image of the photographic object is generated using pixel information from a first pixel among a plurality of pixels possessed by a solid-state imaging element, and an optical flow is derived using pixel information from a second pixel. Thus, even if jitter or defocusing occurs in the image generated using the pixel information from the first pixel due to movement of the photographic object, jitter of the imaging device, etc., the image can be corrected using the optical flow, thereby removing or suppressing the jitter or defocusing. As a result, a clear image (corrected image) is output (displayed, etc.).

[0062] Furthermore, when the first pixel continuously captures moving images at a certain frame rate in order to capture moving images, moving image information exceeding the original frame rate of the first pixel can be generated using the optical flow.

[0063] In addition, in the imaging device of this embodiment,

[0064] The frame rate at which the pixel information is read out from the second pixel may be switched between the second frame rate and the first frame rate.

[0065] According to the above configuration, an image can be generated using pixel information read out from the second pixel at the first frame rate in addition to pixel information from the first pixel, thereby further improving the quality of the generated image.

[0066] In addition, in the imaging device of this embodiment,

[0067] The plurality of second pixels may be arranged at intervals from each other in the pixel array.

[0068] By arranging the second pixels at intervals from each other in this manner, pixel information for deriving the optical flow can be obtained from the entire area of ​​the pixel array, thereby further improving the quality of the corrected image.

[0069] In addition, in the imaging device of this embodiment,

[0070] It is also possible that the first pixels are more than the second pixels.

[0071] According to the above configuration, the number of pixels for generating an image is ensured, and even if the second pixels (pixels for deriving the optical flow) are arranged in the pixel array, degradation in the quality of the generated image can be suppressed.

[0072] In addition, the camera device of this embodiment may further include:

[0073] a first A / D converter to which the pixel information read out from the first pixel is input; and

[0074] a second A / D converter to which the pixel information read out from the second pixel is input,

[0075] A resolution of the second A / D converter is smaller than a resolution of the first A / D converter.

[0076] By reducing the resolution in this manner, it is easy to ensure the processing speed of the second A / D converter, thereby enabling appropriate processing of pixel information read out at the second frame rate (high frame rate).

[0077] In addition, in the imaging device of this embodiment,

[0078] The solid-state imaging element may include a color filter in which red, green, and blue colors are arranged in a predetermined pattern and overlap with the pixel array.

[0079] The second pixel is arranged in the pixel array at a position corresponding to a planned arrangement position of red or blue in the pattern of the color filter.

[0080] Since humans are sensitive to changes in the brightness of green (resolution), by ensuring the number of first pixels of the green filter as in the above-mentioned configuration, even if the number of pixels (first pixels) used for image generation is reduced by configuring the second pixel in the pixel array, the degradation of the image quality of the generated image can be suppressed.

[0081] In this case,

[0082] The portion of the color filter corresponding to the second pixel may be colorless and transparent.

[0083] According to the above configuration, it is possible to suppress a decrease in the amount of light received by the second pixel due to the color filter.

[0084] In addition, the imaging unit of this embodiment includes:

[0085] a solid-state imaging element having a pixel array in which a plurality of pixels are arranged in a matrix; and

[0086] an image generating unit for generating an image based on pixel information from the plurality of pixels,

[0087] The plurality of pixels comprises:

[0088] a plurality of first pixels configured to read out pixel information when light is incident on the pixel array at a first frame rate; and

[0089] a plurality of second pixels configured to read out pixel information when the light is incident on the pixel array at a second frame rate higher than the first frame rate;

[0090] The image generating unit generates and outputs an image based on the pixel information read out at the first frame rate, or corrects the image generated based on the pixel information read out at the first frame rate using the optical flow derived from the pixel information read out at the second frame rate, and outputs the corrected image (corrected image).

[0091] According to the above-mentioned configuration, an image of the photographic object is generated using pixel information from a first pixel among a plurality of pixels possessed by a solid-state imaging element, and an optical flow is derived using pixel information from a second pixel. Thus, even if jitter or defocusing occurs in the image generated using the pixel information from the first pixel due to movement of the photographic object, jitter of the imaging device, etc., the image can be corrected using the optical flow, thereby removing or suppressing the jitter or defocusing, and as a result, a clear image (corrected image) is output.

[0092] Furthermore, when the first pixel continuously captures moving images at a certain frame rate in order to capture moving images, moving image information exceeding the original frame rate of the first pixel can be generated and output using optical flow.

[0093] In addition, in the imaging unit of this embodiment,

[0094] The image generating unit may also restore pixel information of missing coordinates of the first pixel in the pixel array based on pixel information read from the plurality of second pixels, and generate an image based on the restored pixel information and pixel information read from each first pixel.

[0095] In this way, by configuring a position where a second pixel is located in the pixel array, the pixel information of the missing coordinates of the first pixel, that is, the pixel information that should be obtained (read out) from the first pixel when the first pixel is configured at the configuration position of the second pixel, is restored (predicted) and used for image generation, thereby further improving the image quality of the generated image.

[0096] In addition, the photographing method of the present invention includes:

[0097] Reading out pixel information when light is incident on a pixel array at a first frame rate, wherein the pixel array is configured with a plurality of pixels in a matrix, the plurality of pixels comprising: a plurality of first pixels for reading out pixel information when light is incident on the pixel array at the first frame rate; and a plurality of second pixels for reading out pixel information when the light is incident on the pixel array at a second frame rate higher than the first frame rate; and

[0098] outputting an image generated based on the pixel information read out at the first frame rate, or

[0099] Pixel information when the light is incident on the pixel array is read out at the second frame rate, an optical flow is derived from the pixel information read out at the second frame rate, and an image generated based on the pixel information read out at the first frame rate is corrected using the optical flow, and the corrected image is output.

[0100] According to the above method, an image of the photographic object is generated using pixel information from a first pixel among a plurality of pixels, and an optical flow is derived using pixel information from a second pixel. Thus, even if jitter or defocusing occurs in the image generated using the pixel information from the first pixel due to movement of the photographic object, jitter of the photographing device, etc., the image can be corrected using the optical flow, thereby removing or suppressing the jitter or defocusing, and as a result, a clear image (corrected image) is output.

[0101] Furthermore, when the first pixel continuously captures moving images at a certain frame rate in order to capture moving images, moving image information exceeding the original frame rate of the first pixel can be generated and output using optical flow.

[0102] In addition, in the photographing method of this embodiment,

[0103] When outputting an image generated based on the pixel information read out at the first frame rate,

[0104] Pixel information of missing coordinates of a first pixel in the pixel array is restored based on pixel information read from the plurality of second pixels, and the image is generated based on the restored pixel information and pixel information read at the first frame rate.

[0105] In this way, by configuring a position where a second pixel is located in the pixel array, the pixel information of the missing coordinates of the first pixel, that is, the pixel information that should be obtained (read out) from the first pixel when the first pixel is configured at the configuration position of the second pixel, is restored (predicted) and used for image generation, thereby further improving the image quality of the generated image.

[0106] As described above, according to the present embodiment, it is possible to provide an imaging device and an imaging unit capable of clearly capturing a moving subject.

[0107] Hereinafter, one embodiment of the present invention will be described with reference to the drawings.

[0108] The imaging device can capture still images and moving images, and is a digital camera, a smartphone, a tablet device, etc. As shown in FIG1 and FIG2 , the imaging device 1 of this embodiment is a smartphone.

[0109] Specifically, as also shown in FIG3 , the smartphone (camera) 1 includes a solid-state imaging element 2 having a pixel array 21, and a smartphone body (camera body) 3 having an optical system 31 composed of at least one optical element such as a lens. The solid-state imaging element 2 is mounted on the smartphone body 3 so that light passing through the optical system 31 is incident on the pixel array 21. Furthermore, the smartphone 1 includes a control unit 4 connected to the solid-state imaging element 2. The smartphone 1 of this embodiment further includes a nonvolatile memory 101, a working memory 102, an operation unit 103, a display unit 104, a storage medium 105, a connection unit 106, a short-range wireless communication unit 107, a public network connection unit 108, a microphone 109, and a speaker 110.

[0110] Under the control of the control unit 4, the smartphone 1 converts the image of the object (subject) imaged by the optical system 31 of the smartphone body 3 into an electrical signal, performs noise reduction processing, etc., and outputs the digital data as an output image (image data) (for example, displayed on the display unit 104, etc.).

[0111] The solid-state imaging element 2 is an element for imaging a subject, and more specifically, an element for converting an image of the subject formed by the optical system 31 into an electrical signal. The solid-state imaging element 2 of this embodiment is, for example, a CMOS image sensor.

[0112] As also shown in FIG. 4 , the solid-state imaging element 2 includes a pixel array 21 , a color filter 25 superimposed on the pixel array 21 , and an A / D converter 26 that performs A / D conversion on a signal (pixel information) read from each pixel 22 .

[0113] Color filters 25 are used to limit the wavelength of light incident on each pixel 22. Filters that transmit red, green, and blue wavelengths are arranged in a predetermined pattern. In this embodiment, the red, green, and blue filters in color filters 25 are arranged in a Bayer array pattern. However, the arrangement pattern is not limited to a Bayer array of red, green, and blue; other colors or other array patterns are also possible.

[0114] In the pixel array 21, a plurality of pixels 22, each having a photoelectric conversion element FD, are arranged in a matrix (see FIG3 ). Specifically, the pixel array 21 is composed of a plurality of pixels 22 arranged in a matrix. FIG3 shows a portion of the pixel array 21. In the pixel array 21, the arrangement shown in FIG3 is repeated in the row direction (left-right direction in FIG3 ) and the column direction (up-down direction in FIG3 ). Furthermore, in each pixel 22 in FIG3 and FIG4 , a pattern such as a dot is added to indicate the difference in color of the color filter 25 (see FIG4 ).

[0115] The plurality of pixels 22 include: a plurality of first pixels 23 for reading out pixel information (voltage signals corresponding to signal charges generated by the photoelectric conversion elements FD of each pixel 22) at a first frame rate when light passing through the optical system 31 enters the pixel array 21; and a plurality of second pixels 24 for reading out pixel information at a second frame rate higher than the first frame rate when light passing through the optical system 31 enters the pixel array 21. In this embodiment, the first pixels 23 and the second pixels 24 have the same configuration, differing only in the frame rates at which the pixel information is read out.

[0116] In this embodiment, the first frame rate is, for example, 60 fps, and the second frame rate is, for example, 1000 fps. The first frame rate may refer to the readout rate of RGB pixels on the CMOS image sensor, measured in times per second. The second frame rate may refer to the readout rate of EVS pixels on the CMOS image sensor, also measured in times per second. Typically, the second frame rate is much greater than the first frame rate. Furthermore, the frame rates for reading pixel information from each pixel 23, 24 are not limited to these values.

[0117] In the pixel array 21 of this embodiment, the number of first pixels 23 is greater than the number of second pixels 24. For example, the ratio of the number of first pixels 23 to the number of second pixels 24 in the pixel array 21 is 8:1, 16:1, 32:1, or the like.

[0118] The second pixels 24 are spaced apart from each other in the pixel array 21 (see FIG4 ). Specifically, the second pixels 24 are arranged substantially evenly throughout the entire pixel array 21 so that the intervals between adjacent second pixels 24 are the same.

[0119] Furthermore, each second pixel 24 is arranged in the pixel array 21 at a position corresponding to the predetermined position for red or blue in the arrangement pattern of the color filter 25 (in this embodiment, a Bayer array). Furthermore, the portion of the color filter 25 corresponding to the second pixel 24 is colorless and transparent. Specifically, in the color filter 25 of this embodiment, the number of red or blue filters is reduced without reducing the number of green filters in the Bayer array arrangement, and the portion where the red or blue filters are removed is rendered colorless and transparent.

[0120] The A / D converter 26 converts pixel information (analog signals) from the pixel array 21 into digital signals and outputs them to a later stage. In this embodiment, the A / D converter 26 includes a plurality of first A / D converters 261, which receive pixel information read from the first pixels 23, and a plurality of second A / D converters 262, which receive pixel information read from the second pixels 24. These first A / D converters 261 and second A / D converters 262 are arranged for each column (each column of pixels 22) of the pixel array 21, where pixels 22 are arranged in a matrix.

[0121] Each first A / D converter 261 is connected to each first pixel 23 in the column direction of the pixel array 21 via a first signal line 27a, and each second A / D converter 262 is connected to each second pixel 24 in the column direction of the pixel array 21 via a second signal line 27b. In the A / D converter 26 of this embodiment, the resolution of the second A / D converter 262 is lower than that of the first A / D converter 261. For example, the resolution of the second A / D converter 262 of this embodiment is 6 bits, while the resolution of the first A / D converter 261 of this embodiment is 10 bits. In FIG3 , the line connected to each first pixel 23 in the row direction is a first control line 28a, and the line connected to each second pixel 24 in the row direction is a second control line 28b. These control lines 28a and 28b are used for resetting each pixel 23 and 24, selecting a row from which pixel information (signals) is to be read, and for controlling each pixel 23 and 24.

[0122] The control unit 4 controls each component of the smartphone 1 according to input signals or programs. Furthermore, the control unit 4 generates a captured image (an image for external output) based on the signal (pixel information) output from the solid-state imaging element 2 and outputs it to the display unit 104. Alternatively, the control unit that controls each component of the smartphone 1 and the control unit 4 that performs control related to imaging, etc., may be provided separately.

[0123] The control unit 4 includes a processing unit (image generation unit) 41, directly or indirectly connected to the solid-state imaging element 2. This processing unit 41 derives an optical flow from the pixel information read from the second pixels 24, generates an image based on the pixel information read from the first pixels 23, and corrects the generated image using the derived optical flow. The processing unit 41 in this embodiment functionally includes an image generation unit that generates an image based on the pixel information read from the first pixels 23; an optical flow calculation unit that derives the optical flow from the pixel information read from the second pixels 24; and a correction calculation unit that corrects the image using the optical flow.

[0124] Optical flow refers to the movement of optical points between two temporally consecutive image frames in a digital image, observable by projecting the motion of an object, etc., onto the image. For example, it is obtained by vectorizing the differences between feature points (moving points) between two temporally adjacent image frames in a moving image. In the smartphone 1 of this embodiment, the pixel array 21 of the solid-state imaging element 2 performs multiple captures, with the frame rate (second frame rate) for reading pixel information from the second pixels 24 being higher than the frame rate (first frame rate) for reading pixel information from the first pixels 23. The processing unit 41 derives optical flow by calculating, for example, the vectors (movement direction and distance) of each moving point in each of two temporally adjacent pairs of images from the multiple images obtained through this capture (images captured using the second pixels 24).

[0125] In addition, when the operation processing unit 41 derives (calculates) the optical flow from the pixel information read out from the second pixel 24, it produces an event signal quantized into three values ​​of +1 / 0 / -1 based on the difference of continuous images (images based on the pixel information read out from each second pixel 24), and after reducing the data transmission band, uses the produced event signal to derive the optical flow. Among them, +1 refers to the case where the brightness increases by more than a certain proportion, 0 refers to the case where the change in brightness is within a certain range, and -1 refers to the case where the brightness decreases by more than a certain amount. In addition, when the operation processing unit 41 produces an event signal based on the pixel information read out from the second pixel 24, the event signal can be produced based on the brightness signal (Y). In addition, when there is a margin in the data transmission band, the operation processing unit 41 can also directly calculate and derive the optical flow based on the brightness signal with grayscale. In addition, the method of deriving (calculating) the optical flow is not limited to the above method. The optical flow can also be derived using various existing methods (for example, the gradient method, the Lucas-Kanade method, etc.).

[0126] The nonvolatile memory 101 is an electrically erasable and storable nonvolatile memory. The nonvolatile memory 101 of this embodiment stores an OS (Operating System) as basic software executed by the control unit 4 and an application program that realizes application functions in cooperation with the OS.

[0127] The working memory 102 is used as a memory for image display of the display unit 104 , a working area of ​​the control unit 4 , and the like.

[0128] The operation unit 103 is used by a user or the like to input instructions to the smartphone 1. The operation unit 103 of this embodiment includes a power button for turning the smartphone 1 on and off, a touch panel formed on the display unit 104, and the like.

[0129] The display unit 104 displays (outputs to the outside) a captured image (image data), displays characters for operation, and the like.

[0130] The storage medium 105 stores image data (captured images) output from the control unit 4 .

[0131] The connection unit 106 is an interface for connecting to an external device. The smartphone 1 exchanges data with the external device via the connection unit 106 .

[0132] The NFC unit 107 is a communication unit for performing NFC wireless communication and is composed of an antenna for wireless communication, a modulation and demodulation circuit for processing wireless signals, and a communication controller.

[0133] The public network connection unit 108 is an interface for public wireless communications. The smartphone 1 communicates with other devices via this public network connection unit 108 for phone calls. The control unit 4 implements these calls by inputting and outputting audio signals via the microphone 109 and speaker 110. In this embodiment, the public network connection unit 108 is an antenna, and the control unit 4 connects to the public network via this antenna.

[0134] In the smartphone 1 configured in this manner, blurring and defocusing of a captured image can be corrected as follows.

[0135] A subject is photographed using a smartphone 1. In this case, the solid-state imaging device 2 reads pixel information (a first image) from each first pixel 23 at 60 frames per second (fps), while reading pixel information (a second image) from each second pixel 24 at 1000 fps. Specifically, in Figure 3 , the plurality of first pixels 23 corresponding to the first control line 28a are sequentially scanned from top to bottom 60 times per second. Simultaneously, the plurality of second pixels 24 corresponding to the second control line 28b are independently scanned from top to bottom 1000 times per second.

[0136] In addition, in Figure 3, when the multiple second A / D converters 262 arranged in the row direction are not all occupied by pixel information, multiple second control lines 28b can be activated at the same time, and pixel information can be input from the second pixels 24 connected to the multiple second A / D converters 262 via each second signal line 27b.

[0137] After reading out a plurality of second images (images based on the pixel information read out from each second pixel 24) from each second pixel 24 in the pixel array 21, the processing unit 41 derives an optical flow and uses this derived optical flow to correct the corresponding first image (images based on the pixel information read out from each first pixel 23). This corrects for blur and defocus caused by the subject's motion in the first image, resulting in a clear image (corrected image).

[0138] In addition, in the corrected image, the control unit 4 of this embodiment determines the confidence of the captured image corrected using optical flow, and outputs an uncorrected captured image (an image generated using pixel information read from the first pixel 23) when the confidence is low.

[0139] The confidence level is determined by, for example, object recognition based on AI, etc. In this case, for example, when a face is not determined to be a face, the confidence level is determined to be low.

[0140] In addition, the confidence level can also be determined using a second image based on the pixel information read out from the second pixel 24. Specifically, although the second image output from the second pixel 24 has a low resolution and a poor SN ratio, it reflects information that there is no jitter. Therefore, the confidence level of the correction result is determined based on this second image. In addition, the confidence level of the first image corrected using optical flow can be determined, and when the confidence level is low, correction can be performed again using the second image output from the second pixel 24. Specifically, although the second image output from the second pixel 24 has a low resolution and a poor SN ratio, it reflects information that there is no jitter. Therefore, the image information from the first pixel 23 is replaced or mixed based on this image information.

[0141] The smartphone (camera) 1 described above includes: a solid-state imaging element 2 having a pixel array 21 in which a plurality of pixels 22 are arranged in a matrix; and a smartphone body (camera body) 3 having an optical system 31. The solid-state imaging element 2 is mounted on the smartphone body 3 so that light passing through the optical system 31 is incident on the pixel array 21. The plurality of pixels 22 include: a plurality of first pixels 23 for reading pixel information when light enters the pixel array 21 at a first frame rate; and a plurality of second pixels 24 for reading pixel information when light enters the pixel array 21 at a second frame rate higher than the first frame rate. The smartphone body 3 outputs an image generated based on the pixel information read at the first frame rate, or outputs a corrected image obtained by deriving an optical flow from the pixel information read at the second frame rate and using the optical flow to correct an image generated based on the pixel information read at the first frame rate.

[0142] According to the above configuration, an image of the subject is generated using pixel information from the first pixels 23 among the plurality of pixels 22 included in a single solid-state imaging element 2, and optical flow is derived from pixel information from the second pixels 24. Thus, even if the image generated using the pixel information from the first pixels 23 experiences jitter or defocus due to, for example, subject motion or camera shake, the image can be corrected using the optical flow to remove or suppress the jitter or defocus, resulting in a clear image (corrected image) being output.

[0143] Furthermore, when the first pixels 23 continuously capture images at a certain frame rate in order to capture moving images, moving image information exceeding the original frame rate of the first pixels 23 can be generated using optical flows.

[0144] Furthermore, in the smartphone 1 of the present embodiment, the plurality of second pixels 24 are arranged at intervals from one another in the pixel array 21 .

[0145] By arranging the second pixels 24 at intervals from each other in this manner, pixel information for deriving the optical flow can be obtained from the entire area of ​​the pixel array 21 , thereby further improving the quality of the corrected image.

[0146] Furthermore, in the smartphone 1 of this embodiment, the number of first pixels 23 is greater than the number of second pixels 24. Therefore, the number of pixels used to generate an image (the number of first pixels 23) is ensured, and even if second pixels (pixels for deriving optical flow) 24 are arranged in the pixel array 21, degradation in the quality of the generated image can be prevented.

[0147] In addition, the smartphone 1 of this embodiment includes: a first A / D converter, which is input with pixel information read from the first pixel 23; and a second A / D converter, which is input with pixel information read from the second pixel 24, and the resolution of the second A / D converter is smaller than the resolution of the first A / D converter.

[0148] According to the above configuration, it is easy to ensure the processing speed of the second A / D converter, thereby enabling appropriate processing of pixel information read at the second frame rate.

[0149] In addition, in the smartphone 1 of this embodiment, the solid-state imaging element 2 has a color filter 25, which is configured with red, green, and blue in a predetermined pattern and overlaps with the pixel array 21, and the second pixel 24 is configured in the pixel array 21 at a position corresponding to the predetermined configuration position of red or blue in the pattern of the color filter 25.

[0150] Since humans are sensitive to changes in the brightness of green (resolution), by ensuring the number of first pixels 23 overlapped with green filters as in the above-mentioned configuration, even if the number of pixels (first pixels) 23 used to generate an image is reduced by configuring the second pixels 24 in the pixel array 21, the reduction in the quality of the generated image can be suppressed.

[0151] In the smartphone 1 of the present embodiment, the portion of the color filter 25 corresponding to the second pixel 24 is colorless and transparent.

[0152] According to the above configuration, it is possible to suppress a decrease in the amount of light received by the second pixel 24 due to the color filter 25 .

[0153] Furthermore, the imaging device of the present invention is not limited to the above-described embodiments, and various modifications may be implemented without departing from the spirit of the present invention. For example, the configuration of one embodiment may be added to the configuration of another embodiment, or a portion of the configuration of one embodiment may be replaced with the configuration of another embodiment. Furthermore, a portion of the configuration of one embodiment may be deleted.

[0154] The imaging device 1 in the above embodiment is a smartphone, but the configuration is not limited to this. The imaging device 1 may be another device (device having an imaging function) such as a digital camera or a tablet device (mobile terminal).

[0155] In addition, the imaging device 1 of the above embodiment includes a single imaging unit (optical system 31 and solid-state imaging element 2 corresponding to the optical system 31), but is not limited to this configuration. The imaging device 1 may also include multiple imaging units. For example, specifically, the imaging device 1 may include a telephoto imaging unit capable of performing telephoto photography, a wide-angle imaging unit capable of performing wide-angle photography, and an ultra-wide-angle imaging unit capable of performing ultra-wide-angle photography. In this case, the focal length of the telephoto imaging unit is 80 mm or greater when converted to 35 mm film, the focal length of the wide-angle imaging unit is 24 mm when converted to 35 mm film, and the focal length of the ultra-wide-angle imaging unit is 14 mm or less when converted to 35 mm film.

[0156] In this case, the following configuration may be employed: the pixel array 21 of the solid-state imaging element 2 of each imaging unit includes first pixels 23 and second pixels 24, and each imaging unit performs imaging at a first frame rate using the first pixels 23 and at a second frame rate using the second pixels 24. In other words, image generation and optical flow derivation may be performed in each imaging unit.

[0157] Alternatively, for example, a configuration may be employed in which only the solid-state imaging element 2 of the imaging unit with the widest angle of view (in the above example, the ultra-wide-angle imaging unit) among the multiple imaging units has the first pixel 23 and the second pixel 24, while the solid-state imaging elements 2 of the remaining imaging units (in the above example, the wide-angle imaging unit and the telephoto imaging unit) have only the first pixel 23. In this case, when imaging is performed by any of the imaging units, an optical flow is derived from the imaging at the second pixel 24 by the ultra-wide-angle imaging unit, and this optical flow is used to correct the image captured by the first pixel 23 of each imaging unit.

[0158] Furthermore, the imaging device 1 of the above-described embodiment derives optical flow for each image captured and determines whether to use this derived optical flow for image correction (an image generated using pixel information from the first pixels 23) based on the confidence level. However, this configuration is not limited to this. Alternatively, a configuration may be employed in which the photographer or the like manually selects whether to derive optical flow. Alternatively, for example, a configuration may be employed in which the optical flow is automatically switched to be derived when the accumulation time of incident light on the solid-state imaging element 2, which is normally used for imaging, is 1 / 1000 second or longer.

[0159] Furthermore, in the imaging device 1 of the above embodiment, pixel information is read from the second pixels 24 at the second frame rate, but the present invention is not limited to this configuration. The imaging device 1 may also be configured such that the frame rate at which pixel information is read from the second pixels 24 can be switched between the second frame rate and the first frame rate. When an image generated using pixel information from the first pixels 23 is free of jitter or defocus, the quality of the generated image can be further improved by reading pixel information from the second pixels 24 at the first frame rate and using it to generate the image.

[0160] In the above-described embodiment, the processing unit 41 of the imaging device 1 is included in the control unit 4 that controls the imaging device (smartphone) 1, but the present invention is not limited to this configuration. Alternatively, the solid-state imaging element 2 may include the processing unit 41. Alternatively, the solid-state imaging element 2 and the processing unit (image generation unit) 41 may constitute an imaging unit, and this imaging unit may be mounted on the imaging device 1.

[0161] In this imaging unit, the image generation unit can also restore (predict) the pixel information of the missing coordinates of the first pixel 23 in the pixel array 21, that is, the pixel information that should be obtained (read) from the first pixel 23 when the first pixel 23 is configured at the configuration position of the second pixel 24, based on the pixel information read from the plurality of second pixels 24, and generate an image based on the restored pixel information and the pixel information read from each first pixel 23. In this case, when the image generation unit generates the image, the restoration can be performed together with the correction by optical flow, or the restoration can be performed with the correction by optical flow turned off. In addition, the restoration (restoration (calculation, prediction, etc.) of the pixel information in the missing coordinates) can be calculated based on the pixel information read from the first pixels 23 around the coordinates through operations, etc., or can be predicted using AI, etc.

[0162] In this way, at the position where the second pixel 24 is configured in the pixel array 21, the pixel information of the missing coordinates of the first pixel 23, that is, the pixel information that should be obtained (read out) from the first pixel 23 when the first pixel 23 is configured at the configuration position of the second pixel 24, is restored (predicted, etc.) and used for image generation, thereby further improving the image quality of the generated image.

[0163] Furthermore, in the imaging device 1 of the above embodiment, the first pixel 23 and the second pixel 24 of the solid-state imaging element 2 have the same configuration and operation, but this is not the only configuration. The first pixel 23 and the second pixel 24 may also have different configurations and operations. A required characteristic of the second pixel 24 is that it must output a large signal within a short exposure time. Therefore, for example, the conversion gain for converting signal charge into voltage by the second pixel 24 may be designed or set to be greater than the first gain, i.e., the conversion gain of the first pixel 23, but this is not the only configuration.

[0164] To illustrate the present invention, the present invention has been appropriately and fully described above through the embodiments with reference to the accompanying drawings. However, those skilled in the art will recognize that modifications and / or improvements to the above-described embodiments are readily achievable. Therefore, as long as modifications or improvements implemented by those skilled in the art do not depart from the scope of the claims as set forth in the claims, such modifications or improvements should be construed as being included within the scope of the claims.

[0165] Description of reference numerals:

[0166] 1…Smartphone (camera), 2…Solid-state imaging element, 21…Pixel array, 22…Pixel, 23…First pixel, 24…Second pixel, 25…Color filter, 26…A / D converter, 261…First A / D converter, 262…Second A / D converter, 27a…First signal line, 27b…Second signal line, 28a…First control line, 28b…Second control line, 3…Smartphone body, 31…Optical system, 4…Control unit, 41…Processing unit, 101…Nonvolatile memory, 102…Work memory, 103…Operation unit, 104…Display unit, 105…Storage medium, 106…Connector, 107…Near-field wireless communication unit, 108…Public network connection unit, 109…Microphone, 110…Speaker, FD…Photoelectric converter

Claims

1. A photographing device, characterized in that: have: A solid-state imaging element having a pixel array in which a plurality of pixels are arranged in a matrix; and The camera body has an optical system. The solid-state imaging element is mounted on the imaging device body so that light passing through the optical system is incident on the pixel array. The plurality of pixels comprises: a plurality of first pixels, which are used to read out pixel information when the light is incident on the pixel array at a first frame rate; and a plurality of second pixels, which are used to read out pixel information when the light is incident on the pixel array at a second frame rate higher than the first frame rate, The shooting device body outputs an image generated based on the pixel information read out at the first frame rate, or outputs a corrected image, and the corrected image is obtained in the following manner: an optical flow is derived from the pixel information read out at the second frame rate, and the optical flow is used to correct the image generated based on the pixel information read out at the first frame rate, thereby obtaining the corrected image.

2. The photographing device according to claim 1, characterized in that: The frame rate at which the pixel information is read out from the second pixel can be switched between the second frame rate and the first frame rate.

3. The photographing device according to claim 1 or 2, characterized in that: The plurality of second pixels are arranged at intervals from each other in the pixel array.

4. The imaging device according to any one of claims 1 to 3, characterized in that: The first pixels are more than the second pixels.

5. The imaging device according to any one of claims 1 to 4, characterized in that: have: a first A / D converter to which the pixel information read out from the first pixel is input; and a second A / D converter to which the pixel information read out from the second pixel is input, A resolution of the second A / D converter is smaller than a resolution of the first A / D converter.

6. The imaging device according to any one of claims 1 to 5, characterized in that: The solid-state imaging element has a color filter, in which red, green, and blue are arranged in a predetermined pattern and overlap the pixel array. The second pixel is arranged in the pixel array at a position corresponding to a planned arrangement position of red or blue in the pattern of the color filter.

7. The photographing device according to claim 6, characterized in that: A portion of the color filter corresponding to the second pixel is colorless and transparent.

8. A shooting unit, characterized in that: have: A solid-state imaging element having a pixel array in which a plurality of pixels are arranged in a matrix; as well as an image generating unit for generating an image based on pixel information from the plurality of pixels, The plurality of pixels comprises: a plurality of first pixels, which are used to read out pixel information when light is incident on the pixel array at a first frame rate; as well as a plurality of second pixels, which are used to read out pixel information when the light is incident on the pixel array at a second frame rate higher than the first frame rate, The image generating unit generates and outputs an image based on the pixel information read out at the first frame rate, or corrects the image generated based on the pixel information read out at the first frame rate using the optical flow derived from the pixel information read out at the second frame rate, and outputs the corrected image.

9. The photographing unit according to claim 8, characterized in that: The image generation unit restores pixel information of missing coordinates of first pixels in the pixel array based on pixel information read from the plurality of second pixels, and generates an image based on the restored pixel information and pixel information read from each first pixel.

10. A shooting method, characterized in that: include: Reading out pixel information when light is incident on a pixel array at a first frame rate, wherein the pixel array is configured with a plurality of pixels in a matrix, and the plurality of pixels include: a plurality of first pixels for reading out pixel information when light is incident on the pixel array at the first frame rate; and a plurality of second pixels for reading out pixel information when the light is incident on the pixel array at a second frame rate higher than the first frame rate; and outputting an image generated based on the pixel information read out at the first frame rate, or reading out pixel information when the light is incident on the pixel array at the second frame rate, deriving an optical flow from the pixel information read out at the second frame rate, and correcting an image generated based on the pixel information read out at the first frame rate using the optical flow, And output the corrected image.

11. The shooting method according to claim 10, characterized in that: When outputting an image generated based on the pixel information read out at the first frame rate, The pixel information of the missing coordinates of the first pixel in the pixel array is restored based on the pixel information read out from the plurality of second pixels, and the image is generated based on the restored pixel information and the pixel information read out at the first frame rate.

Images