Photographing system, imaging device equipped with the photographing system, and smartphone equipped with the photographing system

Abstract

A photographing system comprising a plurality of photographing units, the plurality of photographing units including a first photographing unit that outputs a first image for external output by photographing a subject, and a second photographing unit that outputs a second image for deriving an optical flow by photographing the subject.

Description

Technical Field

[0001] The present invention relates to a photographing system, an imaging device including the photographing system, and a smartphone including the photographing system.

Background Art

[0002] Conventionally, imaging devices (such as smartphones) capable of photographing photographing subjects such as people, objects, and landscapes have been known. This imaging device includes a photographing unit having an optical system such as a lens and a solid-state imaging device such as an image sensor (Japanese Patent Application Laid-Open No. 2023-1788).

[0003] When photographing a photographing subject with this imaging device, blur or defocus may occur in the obtained photographed image due to the fast movement of the photographing subject or the shake of the imaging device during photographing. In particular, when the frame rate is reduced (that is, the exposure time during photographing is increased) due to a dim surrounding or the like, the blur or defocus in the photographed image becomes prominent.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] Therefore, an object of the present invention is to provide a photographing system capable of clearly photographing a moving imaging object, an imaging device including the photographing system, and a smartphone including the photographing system.

Means for Solving the Problems

[0006] The photographing system according to the present invention includes a plurality of photographing units, The plurality of imaging units include a first imaging unit that outputs a first image for external output by imaging an imaging target, and a second imaging unit that outputs a second image for deriving an optical flow by imaging the imaging target.

[0007] This imaging system includes an arithmetic processing unit connected to each imaging unit, and the arithmetic processing unit may derive the optical flow from the second image output by the second imaging unit and correct the first image output by the first imaging unit with the optical flow.

[0008] Also, in this imaging system, at least one of the plurality of imaging units can be switched between a first imaging mode and a second imaging mode, and the at least one imaging unit in the first imaging mode may perform imaging as the first imaging unit, and the at least one imaging unit in the second imaging mode may perform imaging as the second imaging unit.

[0009] Also, in this imaging system, the second imaging unit may output a plurality of corresponding second images for one first image data output from the first imaging unit.

[0010] Also, in this imaging system, the frame rate of the second imaging unit may be higher than the frame rate of the first imaging unit.

[0011] Also, in this imaging system, the number of pixels of the second image output by the second imaging unit may be smaller than the number of pixels of the first image output by the first imaging unit.

[0012] Also, in this imaging system, The second imaging unit may output an image of a partial area in the captured image as the second image, and make the number of pixels of the output second image smaller than the number of pixels of the first image output by the first imaging unit.

[0013] Also, in the imaging system, The second imaging unit may output, as the second image, an image obtained by temporally and spatially overlapping pixel signals of a plurality of pixels in the captured image.

[0014] Also, in the imaging system, The plurality of imaging units include a first field-of-view imaging unit with a predetermined field of view and a second field-of-view imaging unit with a field of view larger than that of the first field-of-view imaging unit, The first field-of-view imaging unit performs imaging as the first imaging unit, The second field-of-view imaging unit may perform imaging as the second imaging unit.

[0015] Also, in the imaging system, The plurality of imaging units include a telephoto imaging unit capable of telephoto imaging, a wide-angle imaging unit capable of wide-angle imaging, and an ultra-wide-angle imaging unit capable of ultra-wide-angle imaging, When the telephoto imaging unit performs imaging as the first imaging unit, the wide-angle imaging unit performs imaging as the second imaging unit, Or, When the wide-angle imaging unit performs imaging as the first imaging unit, the ultra-wide-angle imaging unit may perform imaging as the second imaging unit.

[0016] Also, in the imaging system, An arithmetic processing unit connected to each imaging unit is provided, The first imaging unit performs multiple exposure for dynamic range expansion imaging, spatial resolution improvement by multiple frame synthesis, or noise reduction by multiple frame synthesis, The arithmetic processing unit may correct the blur in each exposure and correct the shift between multiple exposures using the optical flow.

[0017] Also, in the imaging system, It includes an arithmetic processing unit connected to each imaging unit, The first imaging unit performs multiple exposures for dynamic range expansion imaging, The arithmetic processing unit may use the second image in accordance with the saturation region determination in each exposure of the first image.

[0018] In addition, the imaging device according to the present invention includes any one of the above imaging systems.

[0019] In addition, the smartphone according to the present invention includes any one of the above imaging systems.

[0020] In addition, the imaging method according to the present invention, includes taking a first image for external output of an imaging target and taking a plurality of second images for deriving an optical flow for the imaging target, deriving an optical flow from the plurality of second images, and correcting the first image using the derived optical flow.

[0021] In addition, in the imaging method, the frame rate of taking the second image may be higher than the frame rate of taking the first image.

Brief Description of the Drawings

[0022]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5A

Figure 5B

[0023] The imaging system according to the present invention includes a plurality of imaging units, wherein the plurality of imaging units include a first imaging unit that outputs a first image for external output by imaging an imaging target, and a second imaging unit that outputs a second image for deriving an optical flow by imaging the imaging target.

[0024] According to such a configuration, a first image of the imaging target (a captured image obtained by normal imaging) is obtained by imaging of the first imaging unit, and a second image of the imaging target (an image used for deriving an optical flow) is obtained by imaging of the second imaging unit. Thus, even if blurring or defocusing occurs in the first image obtained by imaging of the first imaging unit due to movement of the imaging target or blurring of the smartphone, etc., by correcting the first image using the optical flow, the blurring or defocusing can be removed or suppressed, and as a result, a clear image can be obtained.

[0025] Further, when the first imaging unit is continuously imaging at a frame rate for video shooting, video information exceeding the original frame rate of the first imaging unit can be generated using the optical flow.

[0026] Further, the imaging system of the present invention includes an arithmetic processing unit connected to each imaging unit, wherein the arithmetic processing unit may derive the optical flow from the second image output by the second imaging unit and correct the first image output by the first imaging unit using the optical flow.

[0027] According to such a configuration, in the imaging system, the first image (captured image) output from the first imaging unit can be corrected using the optical flow corresponding to the first image.

[0028] Also, in the imaging system of the present invention, At least one of the plurality of imaging units is switchable between a first imaging mode and a second imaging mode, The at least one imaging unit in the first imaging mode may perform imaging as the first imaging unit, and the at least one imaging unit in the second imaging mode may perform imaging as the second imaging unit.

[0029] According to such a configuration, the at least one imaging unit can be used as the first imaging unit and as the second imaging unit by switching the imaging mode.

[0030] Also, in the imaging system of the present invention, The second imaging unit may output a plurality of corresponding second images for one first image output from the first imaging unit.

[0031] According to such a configuration, vector information (optical flow) about the movement of the imaging target shown in the first image can be derived (acquired) from the plurality of second images obtained by the second imaging unit.

[0032] In this case, by making the frame rate of the second imaging unit higher than the frame rate of the first imaging unit, the second imaging unit may output a plurality of corresponding second images for one first image output from the first imaging unit.

[0033] Also, in the imaging system of the present invention, The number of pixels of the second image output by the second imaging unit may be smaller than the number of pixels of the first image output by the first imaging unit.

[0034] According to such a configuration, since the capacity of the signal output from the second imaging unit is suppressed, the readout time from an image sensor or the like and the signal processing time in the second imaging unit can be suppressed.

[0035] Also, in the imaging system 1 of the present invention, the second imaging unit may output an image of a partial region in the captured image as the second image, so that the number of pixels of the second image to be output may be smaller than the number of pixels of the first image output by the first imaging unit.

[0036] In this way, by outputting only the pixel signals of the region necessary for deriving the optical flow used for correcting the first image obtained from the first imaging unit among the second images obtained by the second imaging unit, the data amount of the output second image or the like can be suppressed.

[0037] Further, the second imaging unit may output, as the second image, an image in which pixel signals of a plurality of pixels in the captured image are temporally and spatially superimposed.

[0038] According to such a configuration, since the pixel signals of a plurality of pixels in the pixel array of an image sensor or the like are added, averaged, etc. at each pixel of the second image, the signal amount increases. As a result, at each pixel of the second image, the signal amount (luminance information) is sufficiently ensured with respect to noise, and as a result, the signal-to-noise ratio (SN ratio) can be sufficiently improved. That is, in the case of imaging in a dim environment or the like, at each pixel of the captured image by the second imaging unit, although the signal amount (luminance information) is small, by adding, averaging, etc. a plurality of pixel signals to increase the signal amount (luminance information) at each pixel and then using it as the output image (second image), the SN ratio can be sufficiently improved. Thereby, even in imaging in a dim environment, the vector of the movement of the imaging target (optical flow) can be accurately derived.

[0039] Also, in the imaging system of the present invention, The plurality of imaging units includes a first imaging unit with a predetermined angle of view and a second imaging unit with an angle of view larger than that of the first imaging unit. The first imaging unit with the predetermined angle of view performs imaging as the first imaging unit. The second imaging unit with the larger angle of view may perform imaging as the second imaging unit.

[0040] For example, the plurality of imaging units includes a telephoto imaging unit capable of telephoto shooting, a wide-angle imaging unit capable of wide-angle shooting, and an ultra-wide-angle imaging unit capable of ultra-wide-angle shooting. When the telephoto imaging unit performs imaging as the first imaging unit, the wide-angle imaging unit may perform imaging as the second imaging unit. Or, when the wide-angle imaging unit performs imaging as the first imaging unit, the ultra-wide-angle imaging unit may perform imaging as the second imaging unit.

[0041] In addition, the imaging system of the present invention includes an arithmetic processing unit connected to each imaging unit. The first imaging unit performs multiple exposures for dynamic range expansion imaging, spatial resolution improvement by multiple frame synthesis, or noise reduction by multiple frame synthesis. The arithmetic processing unit may correct the blur in each exposure and correct the deviation between multiple exposures using the optical flow.

[0042] According to such a configuration, by correcting the blur in each exposure and correcting the deviation between multiple exposures, a clearer captured image can be obtained.

[0043] In addition, the imaging system of the present invention includes an arithmetic processing unit connected to each imaging unit. The first imaging unit performs multiple exposures for dynamic range expansion imaging. The arithmetic processing unit may use the second image in accordance with the determination of the saturation region in each exposure of the first image.

[0044] According to such a configuration, the saturation region of the first image is corrected, and a clear image can be obtained.

[0045] The imaging device according to the present invention includes any one of the above-described imaging systems.

[0046] Further, the smartphone according to the present invention includes any one of the above-described imaging systems.

[0047] Further, the imaging method according to the present invention includes taking a first image for external output with respect to an imaging target and taking a plurality of second images for deriving an optical flow with respect to the imaging target, deriving an optical flow from the plurality of second images, and correcting the first image using the derived optical flow.

[0048] According to such a configuration, even if blurring or defocusing occurs in the first image obtained by imaging due to the movement of the imaging target or the shake of the smartphone, etc., by correcting the first image using the optical flow, the blurring or defocusing can be removed or suppressed, and as a result, a clear image can be obtained.

[0049] In this case, the frame rate of shooting the second image may be higher than the frame rate of shooting the first image.

[0050] Hereinafter, an embodiment of the present invention will be described with reference to the accompanying drawings.

[0051] As shown in FIG. 1, the imaging system 1 of the present embodiment includes a plurality of imaging units (cameras) 2 and a control unit 3 that controls the plurality of imaging units 2, and can capture still images and moving images. This imaging system 1 is provided in an imaging device such as a digital camera, a smartphone, or a tablet device. The imaging system 1 of the present embodiment is provided in the smartphone 100. In other words, the imaging system 1 of the present embodiment is configured by using the plurality of imaging units (cameras) 2 and the control unit 3 of the smartphone 100.

[0052] Specifically, as shown in FIG. 2, the smartphone 100 includes a plurality of imaging units (cameras) 2. The smartphone 100 also includes a control unit 3 capable of controlling each imaging unit 2. The smartphone 100 further includes a non-volatile memory 101, a working memory 102, an operation unit 103, a display unit 104, a recording medium 105, a connection unit 106, a short-range wireless communication unit 107, a public network connection unit 108, a microphone 109, and a speaker 110. In the smartphone 100 of the present embodiment, as described above, the plurality of imaging units 2 and the control unit 3 constitute the imaging system 1. However, the control unit 3 may be incorporated in any one of the imaging units 2, and the imaging system 1 may be constituted by the plurality of imaging units 2.

[0053] Under the control of the control unit 3, each of the plurality of imaging units 2 converts the image of the imaging target (subject) formed by the optical system 21 included in the imaging unit 2 into an electrical signal, performs noise reduction processing, etc., and outputs the digital data as an output image (image data). Specifically, each imaging unit 2 includes an optical system 21 constituted by at least one optical element such as a lens, and a solid-state imaging device 22 that converts the image of the imaging target formed through the optical system 21 into an electrical signal. This solid-state imaging device 22 is an element for imaging the imaging target and has a pixel array constituted by a plurality of pixels arranged in a matrix. The solid-state imaging device 22 of the present embodiment is, for example, a CMOS image sensor.

[0054] The smartphone 100 of this embodiment includes three imaging units 2. Among these three imaging units 2, the angle of view and the focal length are different from each other. Specifically, these three imaging units 2 are a telephoto imaging unit 2A capable of telephoto shooting, a wide-angle imaging unit 2B capable of wide-angle shooting, and an ultra-wide-angle imaging unit 2C capable of ultra-wide-angle shooting. For example, the focal length of the telephoto imaging unit 2A is 80 mm or more in terms of 35 mm film conversion, the focal length of the wide-angle imaging unit 2B is 24 mm in terms of 35 mm film conversion, and the focal length of the ultra-wide-angle imaging unit 2C is 14 mm or less in terms of 35 mm film conversion. Also, the angle of view (field of view) of the wide-angle imaging unit 2B is larger than that of the telephoto imaging unit 2A, and the angle of view of the ultra-wide-angle imaging unit 2C is larger than that of the wide-angle imaging unit 2B.

[0055] In addition, at least one of the three imaging units 2A, 2B, and 2C is configured to be switchable between a first shooting mode and a second shooting mode. In the smartphone 100 of this embodiment, the wide-angle imaging unit 2B and the ultra-wide-angle imaging unit 2C are configured to be able to switch the shooting mode, and the control unit 3 switches the shooting modes of the wide-angle imaging unit 2B and the ultra-wide-angle imaging unit 2C to the first shooting mode and the second shooting mode, respectively. Here, the first shooting mode is a shooting mode for performing normal shooting, and the second shooting mode is a shooting mode (a mode for causing acquisition shooting) for deriving (acquiring) an optical flow (the motion vector of the shooting target). Note that the telephoto imaging unit 2A always shoots in the first shooting mode. Hereinafter, the imaging units 2A, 2B, and 2C in the first shooting mode may be referred to as the first imaging unit M1, and the imaging units 2B and 2C in the second shooting mode may be referred to as the second imaging unit M2.

[0056] In the imaging unit 2, the frame rate in the first shooting mode (when shooting as the first shooting unit M1) is different from the frame rate in the second shooting mode (when shooting as the second shooting unit M2). Specifically, the second shooting unit M2 outputs a plurality of corresponding image data (second images) for one piece of image data (first image) output from the first shooting unit M1. That is, the frame rate of the second shooting unit M2 is higher than the frame rate of the first shooting unit M1. The frame rate of the first shooting unit M1 in the present embodiment is, for example, 60 fps, and the frame rate of the second shooting unit M2 is, for example, 1000 fps. Note that the frame rates of the respective shooting units M1 and M2 are not limited to these values.

[0057] Also, in the imaging unit 2, the number of pixels of the first image output in the first shooting mode (when shooting as the first shooting unit M1) is different from the number of pixels of the second image output in the second shooting mode (when shooting as the second shooting unit M2). Specifically, the number of pixels of the second image output by the second shooting unit M2 is smaller than the number of pixels of the first image output by the first shooting unit M1. Thereby, the reading time of the captured image in the second shooting unit M2 can be suppressed. Further, since the capacity of the second image output from the second shooting unit M2 is suppressed, the processing amount and processing time of the second image in the control unit 3 and the arithmetic processing unit 31 described later can be suppressed.

[0058] Specifically, the second shooting unit M2 makes the number of pixels (resolution) of the output second image smaller than the number of pixels (resolution) of the first image output by the first shooting unit M1 by temporally and spatially overlapping the pixel signals of a plurality of pixels in the captured image. For example, the second shooting unit M2 suppresses the number of pixels (resolution) by adding the pixel signals of each pixel in the solid-state imaging device 22 (specifically, the pixel array) in the vicinity (temporally and spatially overlapping the pixel signals of adjacent pixels). Thereby, even when the luminance of the shooting target is low, the SN ratio (signal-to-noise ratio) can be increased. The data to be overlapped (added, averaged, etc.) at this time may be digital data or analog data before DA conversion.

[0059] Further, the second imaging unit M2 outputs an image of a partial region Ar (see FIG. 4) in the captured image as a second image, thereby making the number of pixels of the output second image smaller than the number of pixels of the first image output by the first imaging unit M1. This region Ar is a region corresponding to the captured image (angle of view of the telephoto imaging unit 2A) by the telephoto imaging unit 2A that captures as the first imaging unit M1 when the wide-angle imaging unit 2B captures as the second imaging unit M2, and is a region corresponding to the captured image (angle of view of the wide-angle imaging unit 2B) by the wide-angle imaging unit 2B that captures as the first imaging unit M1 when the ultra-wide-angle imaging unit 2C captures as the second imaging unit M2.

[0060] Specifically, in the smartphone 100 of the present embodiment, the number of pixels (resolution) of the first image output from the first imaging unit M1 is, for example, 12.5 Mega pixels, and the number of pixels (resolution) of the second image output from the second imaging unit M2 is, for example, 750000 pixes.

[0061] The control unit 3 controls each part of the smartphone 100 according to the input signals and programs. Further, the control unit 3 generates a captured image (image for external output) from the signals (image data) output from each solid-state imaging device 22 and outputs it to the display unit 104. The control unit 3 of the present embodiment also serves as the control unit of the imaging system 1. Note that the control unit for controlling each part of the smartphone 100 and the control unit 3 of the imaging system 1 may be provided separately.

[0062] This control unit 3 causes one of the plurality of imaging units 2 to perform imaging of an imaging target (normal imaging) as the first imaging unit M1, and controls at least one of the remaining imaging units 2 to perform acquisition imaging for deriving (acquiring) an optical flow of the imaging target as the second imaging unit M2. This control unit 3 has an arithmetic processing unit 31 directly or indirectly connected to each imaging unit 2. This arithmetic processing unit 31 derives an optical flow from the second image (image data) output from the second imaging unit M2 and corrects the first image (image data) output from the first imaging unit M1 with the derived optical flow.

[0063] Here, the optical flow refers to the optical movement of each point between two temporally consecutive image frames that can be observed by projecting the movement of a photographed object or the like onto a digital image. For example, it is a vectorized difference of feature points (each moving point) between two temporally adjacent image frames in a moving image. In the photographing system 1 of the present embodiment, in each photographing unit 2B, 2C (specifically, the CMOS sensors of each photographing unit 2B, 2C), photographing is performed a plurality of times with a frame rate higher than that of normal photographing (i.e., photographing is performed as the second photographing unit M2), and from each of two temporally adjacent second-image pairs in the plurality of second images obtained by this photographing, the operation processing unit 31 calculates the vector (moving direction and distance) of each moving point, and thus the optical flow is derived.

[0064] In addition, the control unit 3 switches the wide-angle photographing unit 2B and the ultra-wide-angle photographing unit 2C to the first photographing mode or the second photographing mode, respectively.

[0065] Specifically, as shown in FIG. 3, when there is an input from the operation unit 103 that the photographing of the object to be photographed (normal photographing) is performed by the telephoto photographing unit 2A, the control unit 3 switches the wide-angle photographing unit 2B to the second photographing mode and causes it to perform photographing as the second photographing unit M2. That is, the control unit 3 causes the telephoto photographing unit 2A to perform normal photographing and output a first image, and causes the wide-angle photographing unit 2B to perform acquisition photographing and output a second image. The telephoto photographing unit 2A performs normal photographing (photographing in the first photographing mode) without switching the photographing mode, and thus photographs as the first photographing unit M1 and outputs a first image.

[0066] In addition, when there is an input from the operation unit 103 that the photographing of the object to be photographed (normal photographing) is performed by the wide-angle photographing unit 2B, the control unit 3 switches the wide-angle photographing unit 2B to the first photographing mode and causes it to perform photographing as the first photographing unit M1, and switches the ultra-wide-angle photographing unit 2C to the second photographing mode and causes it to perform photographing as the second photographing unit M2. That is, the control unit 3 causes the wide-angle photographing unit 2B to perform normal photographing and output a first image, and causes the ultra-wide-angle photographing unit 2C to perform acquisition photographing and output a second image.

[0067] When the arithmetic processing unit 31 derives (calculates) the optical flow from the second image output from the second imaging unit M2, it creates an event signal quantized into three values of +1 / 0 / -1 from the difference between consecutive images (second images), reduces the data transmission bandwidth, and then derives the optical flow using the created event signal. Here, +1 means the case where the luminance increases by a certain ratio or more, 0 means the case where the change in luminance is within a certain range, and -1 means the case where the luminance decreases by a certain amount or more. Also, when the arithmetic processing unit 31 creates an event signal from the second image, it may create an event signal from each of R, G, and B, or it may create an event signal based on the luminance signal (Y). If there is a margin in the data transmission bandwidth, the arithmetic processing unit 31 may directly calculate and derive the optical flow from the luminance signal with harmony. Also, the method for deriving (calculating) the optical flow is not limited to the above method. For the derivation of the optical flow, various conventional methods (for example, the gradient method, the Lucas-Kanade method, etc.) may be used.

[0068] The non-volatile memory 101 is a non-volatile memory that can be electrically erased and recorded. The non-volatile memory 101 of the present embodiment records an OS (operating system), which is basic software executed by the control unit 3, and an application that realizes application functions in cooperation with this OS.

[0069] The working memory 102 is used as an image display memory for the display unit 104 and a working area of the control unit 3, etc.

[0070] The operation unit 103 is used for a user or the like to input an instruction to the smartphone 100. The operation unit 103 of the present embodiment includes a power button for instructing ON / OFF of the power of the smartphone 100 and a touch panel formed on the display unit 104, etc.

[0071] The display unit 104 performs display (output to the outside) of the captured image (image data) and character display for operation, etc.

[0072] The recording medium 105 records the image data output from the photographing unit 2.

[0073] The connection unit 106 is an interface for connecting to an external device. Through this connection unit 106, the smartphone 100 exchanges data with the external device.

[0074] The short-range wireless communication unit 107 is a communication unit for performing short-range wireless communication. The short-range wireless communication unit 107 is composed of an antenna for wireless communication, a modulation / demodulation circuit for processing wireless signals, and a communication controller.

[0075] The public network connection unit 108 is an interface for performing public wireless communication. Through this public network connection unit 108, the smartphone 100 communicates with other devices for a call. At this time, the control unit 3 realizes the call by inputting and outputting voice signals via the microphone 109 and the speaker 110. The public network connection unit 108 of the present embodiment is an antenna, and the control unit 3 connects to the public network through this antenna.

[0076] In the smartphone 100 configured as described above, blurring and defocusing of a photographed image can be corrected as follows.

[0077] When a photographer or the like switches the photographing unit 2 used for photographing to the wide-angle photographing unit 2B by input from the operation unit 103, for example, the control unit 3 switches the wide-angle photographing unit 2B to the first photographing mode and switches the ultra-wide-angle photographing unit 2C to the second photographing mode. As a result, the wide-angle photographing unit 2B photographs as the first photographing unit M1, and the ultra-wide-angle photographing unit 2C photographs as the second photographing unit M2.

[0078] Subsequently, the imaging unit 2 of the imaging system 1 (smartphone 100) captures an object to be imaged. In the imaging system 1 of the present embodiment, for example, when the first imaging unit M1 (wide-angle imaging unit 2B) captures a captured image (first image) at 60 frames per second (fps) / 12.5 Mega pixels, the second imaging unit M2 (ultra-wide-angle imaging unit 2C) captures a captured image (second image) for deriving (acquiring) an optical flow at 1000 fps / 75000 pixels.

[0079] At this time, the control unit 3 adjusts the focus position of the second imaging unit M2 based on the focus information of the first imaging unit M1, that is, makes the foci of both imaging units M1 and M2 coincide.

[0080] In addition, since the peripheral portion of the captured image obtained from the second imaging unit M2 (ultra-wide-angle imaging unit 2C) is unnecessary for deriving the optical flow of the captured image (first image) output from the first imaging unit M1 (wide-angle imaging unit 2B), the control unit 3 limits the readout area Ar of the captured image obtained from the second imaging unit M2 in consideration of the angle of view and parallax of the first imaging unit M1, thereby outputting the second image at a higher speed.

[0081] At this time, the limitation of the readout area Ar by the control unit 3 is performed based on the focus information of the first imaging unit M1 (in the example of the present embodiment, the wide-angle imaging unit 2B). Note that the limitation of the readout area Ar is performed in consideration of the large parallax when the in-focus object to be imaged is close.

[0082] Alternatively, a maximum area assuming the parallax from infinity to the closest subject distance from the beginning may be fixedly cut out, and the necessary area may be extracted and used on the processing side as appropriate.

[0083] When the second imaging unit M2 outputs a plurality of second images, the arithmetic processing unit 31 derives an optical flow and corrects the corresponding first image using the derived optical flow. As a result, blurring and defocusing caused by the movement of the object to be imaged in the first image are corrected, and a clear image is obtained.

[0084] Still, in the corrected first image of this embodiment, the control unit 3 determines the reliability of the captured image corrected using the optical flow, and outputs a captured image (the first image as it is output from the first imaging unit M1) without performing correction when the reliability is low.

[0085] This determination of reliability uses, for example, object recognition by AI or the like. In this case, for example, when a human face is not determined to be a human face, it is determined that the reliability is low.

[0086] Also, the second image output from the second imaging unit M2 may be used for the determination of reliability. Specifically, in the second image output from the second imaging unit M2, although the resolution is low and the signal-to-noise ratio is poor, information that is not blurred is reflected, so the reliability of the correction result is determined based on this second image. Still, when the reliability of the first image corrected using the optical flow is determined to be low, correction may be performed again using the second image output from the second imaging unit M2. Specifically, in the second image output from the second imaging unit M2, although the resolution is low and the signal-to-noise ratio is poor, information that is not blurred is reflected, so the image information from the first imaging unit M1 is replaced or blended based on that image information.

[0087] The imaging system 1 of the smartphone 100 described above includes a plurality of imaging units 2. And the plurality of imaging units 2 include a first imaging unit M1 that outputs a first image for external output by imaging a subject, and a second imaging unit M2 that outputs a second image for deriving an optical flow by imaging the subject.

[0088] According to such a configuration, a first image of a subject (a captured image obtained by normal shooting) is obtained by shooting with the first shooting unit M1, and a second image of the subject (an image used for deriving an optical flow) is obtained by shooting with the second shooting unit M2. Thus, even if there is blur or defocus caused by the movement of the subject or the shake of the smartphone in the first image obtained by shooting with the first shooting unit M1, by correcting the first image using the optical flow, the blur or defocus can be removed or suppressed, and as a result, a clear image can be obtained.

[0089] In addition, when the first shooting unit M1 is continuously shooting at a frame rate for video shooting, video information exceeding the original frame rate of the first shooting unit M1 can be generated using the optical flow.

[0090] In addition, when the first shooting unit M1 is performing multiple exposures for dynamic range expansion shooting, spatial resolution improvement by multi-frame synthesis (Multi-Frame Superresolution: MFNR), or noise reduction by multi-frame synthesis (Multi-Frame Noise Reduction: MFNR), not only blur correction of each exposure (captured images in each exposure) but also misalignment correction between multiple exposures (between captured images in each exposure) can be performed using the optical flow.

[0091] In addition, when the first shooting unit M1 is performing dynamic range expansion shooting by multi-frame synthesis, since the second shooting unit M2 is shooting at a high frame rate, the second image can be used in accordance with the saturation region determination in each exposure of the first image, taking advantage of the characteristic that the signal is less likely to saturate even for high-brightness subjects.

[0092] In addition, the imaging method using the imaging system 1 of the smartphone 100 includes capturing a first image for external output with respect to the imaging subject and capturing a plurality of second images for deriving an optical flow with respect to the imaging subject, deriving an optical flow from the plurality of second images, and correcting the first image using the derived optical flow. As a result, even if blurring or defocusing occurs in the first image obtained by imaging due to movement of the imaging subject or blurring of the smartphone, etc., by correcting the first image using the optical flow, the blurring or defocusing can be removed or suppressed, and as a result, a clear image can be obtained.

[0093] In addition, the imaging system 1 of the present embodiment includes an arithmetic processing unit 31 connected to each imaging unit 2. The arithmetic processing unit 31 derives an optical flow from the second image output by the second imaging unit M2 and corrects the first image output by the first imaging unit M1 using the optical flow.

[0094] According to such a configuration, in the imaging system 1, the first image (captured image) output from the first imaging unit M1 can be corrected using the optical flow corresponding to the first image. Note that in the smartphone 100 of the present embodiment, correction using the optical flow is possible for both still images and moving images.

[0095] In addition, in the imaging system 1 of the present embodiment, at least one of the plurality of imaging units 2, i.e., the imaging units 2B and 2C, can be switched between a first imaging mode and a second imaging mode. When in the first imaging mode, the at least one imaging unit 2B and 2C performs imaging as the first imaging unit M1, and when in the second imaging mode, the at least one imaging unit 2B and 2C performs imaging as the second imaging unit M2.

[0096] According to such a configuration, at least one imaging unit 2 (in the example of the present embodiment, the wide-angle imaging unit 2B and the ultra-wide-angle imaging unit 2C) can be used as the first imaging unit M1 and the second imaging unit M2 by switching the imaging mode.

[0097] Also, in the imaging system 1 of the present embodiment, the second imaging unit M2 outputs a plurality of corresponding second images for one first image output from the first imaging unit M1. According to this configuration, vector information (optical flow) about the movement of the imaging target shown in the first image can be derived (acquired) from the plurality of second images obtained by the second imaging unit M2.

[0098] In the imaging system 1 of the present embodiment, by setting the frame rate of the second imaging unit M2 higher than the frame rate of the first imaging unit M1, the second imaging unit M2 outputs a plurality of corresponding second images for one first image output from the first imaging unit M1.

[0099] At this time, by making the number of pixels of the second image output by the second imaging unit M2 smaller than the number of pixels of the first image output by the first imaging unit M1 to suppress the signal capacity, the readout time from an image sensor or the like and the signal processing time in the second imaging unit M2 can be suppressed.

[0100] Also, in the imaging system 1 of the present embodiment, the angle of view of the second imaging unit M2 is larger than the angle of view of the first imaging unit M1. Therefore, the second imaging unit M2 outputs an image of a partial region Ar in the captured image as the second image, thereby making the number of pixels of the output second image smaller than the number of pixels of the first image output by the first imaging unit M1. In this way, by outputting only the pixel signals of the region Ar necessary for deriving the optical flow used for correcting the first image obtained from the first imaging unit M1 among the second images obtained by the second imaging unit M2, the data amount of the output second image and the like can be suppressed.

[0101] Also, the second imaging unit M2 outputs, as the second image, an image in which the pixel signals of a plurality of pixels in the captured image are superposed temporally and spatially (the pixel signals of neighboring pixels are added, averaged, etc.).

[0102] According to such a configuration, since the pixel signals of neighboring pixels in a pixel array such as an image sensor are added, averaged, etc. for each pixel of the second image, the signal amount increases. As a result, for each pixel of the second image, a sufficient signal amount (luminance information) is ensured against noise, and consequently, the signal-to-noise ratio can be sufficiently improved. That is, in the case of shooting in a dim environment or the like, in each pixel of the captured image by the second shooting unit M2, although the signal amount (luminance information) is small, by adding and averaging a plurality of pixel signals to increase the signal amount (luminance information) in each pixel and then using it as the output image (second image), the signal-to-noise ratio can be sufficiently improved. Thereby, even in shooting in a dim environment, the vector of the movement of the shooting target (optical flow) can be accurately derived.

[0103] Further, in the shooting system 1 of the present embodiment, the plurality of shooting units 2 include a first angle-of-view shooting unit with a predetermined angle of view (in the example of the present embodiment, a telephoto shooting unit 2A and a wide-angle shooting unit 2B), and a second angle-of-view shooting unit with an angle of view larger than that of the first angle-of-view shooting units 2A and 2B (in the example of the present embodiment, a wide-angle shooting unit 2B and an ultra-wide-angle shooting unit 2C). The first angle-of-view shooting units 2A and 2B perform shooting (normal shooting) as the first shooting unit M1, and the second angle-of-view shooting units 2B and 2C perform shooting (shooting for deriving (acquiring) an optical flow) as the second shooting unit M2.

[0104] Specifically, the plurality of shooting units 2 include a telephoto shooting unit 2A capable of telephoto shooting, a wide-angle shooting unit 2B capable of wide-angle shooting, and an ultra-wide-angle shooting unit 2C capable of ultra-wide-angle shooting. When the telephoto shooting unit 2A performs shooting as the first shooting unit M1, the wide-angle shooting unit 2B performs shooting as the second shooting unit M2, or when the wide-angle shooting unit 2B performs shooting as the first shooting unit M1, the ultra-wide-angle shooting unit 2C performs shooting as the second shooting unit M2.

[0105] Furthermore, the photographing system 1 of the present invention, the imaging device including the photographing system 1, and the smartphone 100 including the photographing system 1 are not limited to the above-described embodiments, and various modifications can be made without departing from the gist of the present invention. For example, the configuration of one embodiment can be added to the configuration of another embodiment, and a part of the configuration of one embodiment can be replaced with the configuration of another embodiment. Further, a part of the configuration of one embodiment can be deleted.

[0106] The photographing system 1 of the above-described embodiment is provided in the smartphone 100, but is not limited to this configuration. The photographing system 1 may be provided in other imaging devices such as a digital camera and a tablet device (portable terminal).

[0107] Also, the above-described photographing system 1 includes three photographing units 2A, 2B, and 2C, but is not limited to this configuration. The photographing system 1 may include two photographing units 2, or may include four or more photographing units 2.

[0108] Also, in the photographing system 1 of the above-described embodiment, the angle of view and the focal length are different from each other in the plurality of photographing units 2, but are not limited to this configuration. In the plurality of photographing units 2, the angle of view and the focal length of all the photographing units 2 may be the same, or the angle of view and the focal length of some of the photographing units 2 may be the same.

[0109] Also, in the photographing system 1 of the above-described embodiment, the plurality of photographing units 2 include the photographing units 2B and 2C that can be switched between the first photographing mode and the second photographing mode, but are not limited to this configuration. The plurality of photographing units 2 may include a photographing unit that functions only as the first photographing unit M1 and a photographing unit that functions only as the second photographing unit M2. That is, in the plurality of photographing units 2, the photographing unit 2 that functions as the first photographing unit M1 and the photographing unit 2 that functions as the second photographing unit M2 may be fixed.

[0110] In addition, although the imaging system 1 of the above embodiment includes an arithmetic processing unit 31 that derives an optical flow from the second image (image data) output from the second imaging unit M2 and corrects the first image (image data) output from the first imaging unit M1 using the derived optical flow, the present invention is not limited to this configuration. The imaging system 1 may be configured such that it does not include the arithmetic processing unit 31, and the first image is corrected or the like using the arithmetic processing unit 31 provided in another device or the like that can be connected to the imaging system 1.

[0111] In addition, in the imaging system 1 of the above embodiment, when the imaging unit 2 (imaging unit 2 that outputs the first image by imaging) used for normal imaging is switched to the telephoto imaging unit 2A or the wide-angle imaging unit 2B, the wide-angle imaging unit 2B or the ultra-wide-angle imaging unit 2C is automatically selected as the imaging unit 2 (imaging unit 2 that outputs the second image by imaging) for deriving the optical flow. However, the present invention is not limited to this configuration. The configuration may be such that the photographer or the like manually selects whether to perform imaging for deriving the optical flow. Further, for example, when the accumulation time of the incident light in the solid-state imaging device 22 for normal imaging is 1 / 1000 second or greater, the configuration may be such that the imaging for deriving the optical flow is automatically switched to be performed.

[0112] In addition, in the imaging system 1 of the above embodiment, in the two imaging units 2 with different angle of views, the imaging unit 2 with the smaller angle of view is configured to perform imaging as the first imaging unit M1, and the imaging unit 2 with the larger angle of view is configured to perform imaging as the second imaging unit M2. However, the present invention is not limited to this configuration. The configuration may be such that the imaging unit 2 with the larger angle of view performs imaging as the first imaging unit M1, and the imaging unit 2 with the smaller angle of view performs imaging as the second imaging unit M2. Even with this configuration, the optical flow can be derived within the range of the second image obtained from the imaging unit 2 with the smaller angle of view, and correction using the optical flow can be performed in the range (such as the region excluding the peripheral portion) corresponding to the second image in the first image obtained from the imaging unit 2 with the larger angle of view.

[0113] Further, the imaging system 1 may be configured such that video frame interpolation is performed on the output of the first imaging unit M1 using the optical flow derived from the captured image (second image) of the second imaging unit M2.

[0114] In the imaging system 1 of the above embodiment, although there is one imaging unit 2 that captures images as the second imaging unit M2 for deriving the optical flow with respect to the captured image (first image) of one first imaging unit M1, the present invention is not limited to this configuration. In the imaging system 1 including three or more imaging units 2, there may be a plurality of imaging units 2 that capture images as the second imaging unit M2 for deriving the optical flow with respect to the captured image (first image) of one first imaging unit M1. According to such a configuration, a more accurate optical flow in terms of space and time can be obtained by combining and using a plurality of pieces of information (outputs from the second imaging unit M2).

[0115] Further, when the ultra-wide-angle imaging unit 2C is used for obtaining the optical flow (i.e., used as the second imaging unit M2) and correcting the video captured by the wide-angle imaging unit 2B (first imaging unit M1), as shown in FIG. 5B, not all of the image captured by the ultra-wide-angle imaging unit 2C is necessary, and only the portion corresponding to the image (field of view) captured by the wide-angle imaging unit 2B is required. Here, FIG. 5A shows the image captured by the wide-angle imaging unit 2B, and FIG. 5B shows the image captured by the ultra-wide-angle imaging unit 2C.

[0116] In such a case, for block readout from a CMOS image sensor or the like, for example, in the example shown in FIG. 5B, block readout is performed by skipping the first 500 lines and the last 500 lines out of a total of 3000 lines, and the first 500 columns and the last 500 columns out of a total of 4000 columns, thereby reducing the number of rows and columns to be accessed and reducing the amount of data read from the ultra-wide-angle imaging unit 2C, and increasing the frame rate for image acquisition. As a result, in optical flow acquisition, the temporal resolution of the optical flow can be increased, or when the temporal resolution is kept constant, a pause period can be provided between frames to stop the sensor, and the effect that an equivalent optical flow can be obtained with lower power can be obtained.

[0117] In order to represent the present invention, the present invention has been appropriately and fully described through embodiments with reference to the drawings above. However, it should be recognized that those skilled in the art can easily make changes and / or improvements to the above embodiments. Therefore, as long as the changes or improvements implemented by those skilled in the art do not depart from the scope of the claims described in the claims, the changes or improvements are construed to be included in the scope of the claims.

Explanation of Reference Numerals

[0118] 1... Imaging system, 2... Imaging unit, 2A... Telephoto imaging unit, 2B... Wide-angle imaging unit, 2C... Ultra-wide-angle imaging unit, 21... Optical system, 22... Solid-state imaging device, 3... Control unit, 31... Arithmetic processing unit, 100... Smartphone, 101... Non-volatile memory, 102... Working memory, 103... Operation unit, 104... Display unit, 105... Recording medium, 106... Connection unit, 107... Short-range wireless communication unit, 108... Public network connection unit, 109... Microphone, 110... Speaker, Ar... Region corresponding to the captured image of the first imaging unit, M1... First imaging unit, M2... Second imaging unit

Claims

1. Comprising a plurality of imaging units, The plurality of imaging units include a first imaging unit that outputs a first image for external output by imaging an imaging target, and a second imaging unit that outputs a second image for deriving an optical flow by imaging the imaging target, and an imaging system.

2. Comprising an arithmetic processing unit connected to each imaging unit, The arithmetic processing unit derives the optical flow from the second image output by the second imaging unit, and corrects the first image output by the first imaging unit with the optical flow. The imaging system according to claim 1.

3. At least one of the plurality of imaging units is switchable between a first imaging mode and a second imaging mode, The at least one imaging unit in the first imaging mode performs imaging as the first imaging unit, and the at least one imaging unit in the second imaging mode performs imaging as the second imaging unit. The imaging system according to claim 1 or 2.

4. The second imaging unit outputs a plurality of corresponding second images for one first image output from the first imaging unit. The imaging system according to any one of claims 1 to 3.

5. The frame rate of the second imaging unit is higher than the frame rate of the first imaging unit. The imaging system according to claim 4.

6. The number of pixels of the second image output by the second imaging unit is smaller than the number of pixels of the first image output by the first imaging unit. The imaging system according to any one of claims 1 to 5.

7. The second imaging unit outputs an image of a partial region in the captured image as the second image, so that the number of pixels of the output second image is smaller than the number of pixels of the first image output by the first imaging unit. The imaging system according to claim 6.

8. The second imaging unit outputs, as the second image, an image in which pixel signals of a plurality of pixels are temporally and spatially superimposed. The imaging system according to any one of claims 1 to 6.

9. The plurality of imaging units include a first field-of-view imaging unit with a predetermined field of view, and a second field-of-view imaging unit with a field of view larger than that of the first field-of-view imaging unit, The first field-of-view imaging unit performs imaging as the first imaging unit, The second field-of-view imaging unit performs imaging as the second imaging unit. The imaging system according to any one of claims 1 to 7.

10. The plurality of imaging units include a telephoto imaging unit capable of telephoto imaging, a wide-angle imaging unit capable of wide-angle imaging, and an ultra-wide-angle imaging unit capable of ultra-wide-angle imaging. When the telephoto imaging unit performs imaging as the first imaging unit, the wide-angle imaging unit performs imaging as the second imaging unit. Or When the wide-angle imaging unit performs imaging as the first imaging unit, the ultra-wide-angle imaging unit performs imaging as the second imaging unit. The imaging system according to any one of claims 1 to 8.

11. It includes an arithmetic processing unit connected to each imaging unit. The first imaging unit performs multiple exposures for dynamic range expansion imaging, spatial resolution improvement by multiple frame synthesis, or noise reduction by multiple frame synthesis. The arithmetic processing unit corrects the blur in each exposure and corrects the deviation between multiple exposures using the optical flow. The imaging system according to any one of claims 1 to 10.

12. It includes an arithmetic processing unit connected to each imaging unit. The first imaging unit performs multiple exposures for dynamic range expansion imaging. The arithmetic processing unit uses the second image in accordance with the saturation region determination in each exposure of the first image. The imaging system according to any one of claims 1 to 10.

13. An imaging device comprising the imaging system according to any one of claims 1 to 12.

14. A smartphone comprising the imaging system according to any one of claims 1 to 12.

15. Taking a first image for external output with respect to an imaging object and taking a plurality of second images for deriving an optical flow with respect to the imaging object. Deriving an optical flow from the plurality of second images. Correcting the first image using the derived optical flow. An imaging method comprising:

16. The frame rate of taking the second image is higher than the frame rate of taking the first image. The imaging method according to claim 15.

Images