Imaging system, imaging device provided with imaging system, and smartphone provided with imaging system

Abstract

An imaging system includes a plurality of imaging units including a first imaging unit that outputs a first image for external output by imaging an object to be imaged, and a second imaging unit that outputs a second image for deriving an optical flow by imaging the object to be imaged.

Description

Photographic system, camera having a photographic system, and smartphone having a photographic system Technical Field

[0001] The present invention relates to a photographic system, a shooting device equipped with the photographic system, and a smart phone equipped with the photographic system. Background Art

[0002] Cameras (such as smartphones) capable of capturing images of subjects such as people, objects, and landscapes have long been known. These cameras include an imaging unit comprising an optical system such as a lens and a solid-state imaging element such as an image sensor (Japanese Patent Application Laid-Open No. 2023-1788).

[0003] When photographing a subject using such a camera, the subject's rapid movement or the camera's shaking during photography may cause the resulting image to be jittery or out of focus. This jitter or out of focus may be more noticeable in the captured image, particularly when the frame rate is reduced (i.e., the exposure time is increased) due to factors such as dark surroundings.

[0004] Prior art literature

[0005] Patent Literature

[0006] Patent Document 1: Japanese Patent Application Laid-Open No. 2023-1788.

[0007] Summary of the Invention

[0008] Problems to be solved by the invention

[0009] Therefore, an object of the present invention is to provide a photography system capable of clearly photographing a moving subject, a photography device including the photography system, and a smartphone including the photography system.

[0010] Solutions for solving problems

[0011] The photographic system of the present invention,

[0012] Equipped with multiple photography departments,

[0013] The plurality of imaging units include a first imaging unit that outputs a first image for external output by imaging a subject, and a second imaging unit that outputs a second image for deriving an optical flow by imaging the subject.

[0014] The photography system

[0015] It has a calculation processing unit connected to each photographic unit.

[0016] The arithmetic processing unit may derive the optical flow from the second image output by the second imaging unit, and correct the first image output by the first imaging unit using the optical flow.

[0017] In addition, in the photography system,

[0018] The control unit may switch at least one of the plurality of photographing units between a first photographing mode and a second photographing mode.

[0019] The at least one photographing unit in the first photographing mode may perform photographing as the first photographing unit, and the at least one photographing unit in the second photographing mode may perform photographing as the second photographing unit.

[0020] In addition, in the photography system,

[0021] The second imaging unit may output a plurality of second images corresponding to one piece of first image data output by the first imaging unit.

[0022] In addition, in the photography system,

[0023] The frame rate of the second photographing unit may be greater than the frame rate of the first photographing unit.

[0024] In addition, in the photography system,

[0025] The number of pixels of the second image output by the second imaging unit may be smaller than the number of pixels of the first image output by the first imaging unit.

[0026] In addition, in the photography system,

[0027] The second photographing unit may output an image of a partial area in the photographed image as the second image, so that the number of pixels of the output second image is smaller than the number of pixels of the first image output by the first photographing unit.

[0028] In addition, in the photography system,

[0029] The second imaging unit may output, as the second image, an image in which pixel signals of a plurality of pixels in a captured image are temporally and spatially superimposed.

[0030] In addition, in the photography system,

[0031] The plurality of photographing units include a first angle-of-view photographing unit with a predetermined angle of view and a second angle-of-view photographing unit with a larger angle of view than the first angle-of-view photographing unit.

[0032] The first viewing angle photographing unit can be used as the first photographing unit to take photos.

[0033] The second-angle-of-view photographing unit can perform photographing as the second photographing unit.

[0034] In addition, in the photography system,

[0035] The plurality of photographing units include a telephoto photographing unit capable of performing telephoto photography, a wide-angle photographing unit capable of performing wide-angle photography, and an ultra-wide-angle photographing unit capable of performing ultra-wide-angle photography.

[0036] When the telephoto photography unit functions as the first photography unit for photography, the wide-angle photography unit functions as the second photography unit for photography.

[0037] or,

[0038] When the wide-angle photographing unit performs the photographing as the first photographing unit, the ultra-wide-angle photographing unit can perform the photographing as the second photographing unit.

[0039] In addition, in the photography system,

[0040] It has a calculation processing unit connected to each photographic unit.

[0041] The first photographing unit performs multiple exposures for dynamic range amplification photography, spatial resolution improvement based on multi-frame synthesis, or noise reduction based on multi-frame synthesis.

[0042] The arithmetic processing unit can use the optical flow to correct the jitter in each exposure and simultaneously correct the deviation between multiple exposures.

[0043] In addition, in the photography system,

[0044] It has a calculation processing unit connected to each photographic unit.

[0045] The first photographing unit performs multiple exposures for dynamic range amplification photography.

[0046] The arithmetic processing unit may use the second image to determine a saturated area in respective exposures in conjunction with the first image.

[0047] Furthermore, the imaging device of the present invention includes any of the above-mentioned imaging systems.

[0048] Furthermore, the smartphone of the present invention includes any of the above-mentioned photography systems.

[0049] Furthermore, the photographing method of the present invention comprises:

[0050] Taking a first image of a photographic subject for external output, and taking a plurality of second images of the photographic subject for deriving an optical flow;

[0051] deriving an optical flow from the plurality of second images;

[0052] The first image is corrected using the derived optical flow.

[0053] In addition, in the photography method,

[0054] The frame rate of the photography of the second image may be higher than the frame rate of the photography of the first image. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] FIG1 is a functional block diagram of a smartphone according to this embodiment.

[0056] FIG. 2 is a diagram showing the smartphone as viewed from the imaging unit side.

[0057] FIG. 3 is a diagram for explaining switching between the first imaging unit and the second imaging unit in each imaging unit.

[0058] FIG. 4 is a diagram illustrating correction of a photographed image using optical flow in the photographic system included in the smartphone.

[0059] FIG. 5A is a schematic diagram illustrating an example of a first image.

[0060] FIG. 5B is a schematic diagram for explaining block readout, and is a schematic diagram showing a range of block readout in the second image. DETAILED DESCRIPTION

[0061] The photographic system of the present invention,

[0062] Equipped with multiple photography departments,

[0063] The plurality of imaging units include a first imaging unit that outputs a first image for external output by imaging a subject, and a second imaging unit that outputs a second image for deriving an optical flow by imaging the subject.

[0064] According to the above configuration, a first image of the photographic subject (the photographic image obtained by normal photography) is obtained by the first photographing unit, while a second image of the photographic subject (the image used for optical flow derivation) is obtained by the second photographing unit. Thus, even if the first image obtained by the first photographing unit experiences jitter or defocus due to, for example, movement of the photographic subject or shaking of the smartphone, the jitter or defocus can be removed or suppressed by correcting the first image using optical flow, resulting in a clear image.

[0065] Furthermore, when the first imaging unit continuously captures images at a certain frame rate for capturing moving images, it is possible to generate moving image information at a frame rate exceeding the original frame rate of the first imaging unit using optical flow.

[0066] In addition, the photographic system of the present invention,

[0067] It has a calculation processing unit connected to each photographic unit.

[0068] The arithmetic processing unit may derive the optical flow from the second image output by the second imaging unit, and correct the first image output by the first imaging unit using the optical flow.

[0069] According to the above configuration, in the photographing system, the first image (photographed image) output by the first photographing unit can be corrected using the optical flow corresponding to the first image.

[0070] In addition, in the photographic system of the present invention,

[0071] At least one of the plurality of photographing units can be switched between a first photographing mode and a second photographing mode.

[0072] The at least one photographing unit in the first photographing mode may perform photographing as the first photographing unit, and the at least one photographing unit in the second photographing mode may perform photographing as the second photographing unit.

[0073] According to the above configuration, the at least one photographing unit can be used as the first photographing unit and the second photographing unit by switching the photographing mode.

[0074] In addition, in the photographic system of the present invention,

[0075] The second imaging unit may output a plurality of second images corresponding to one first image output from the first imaging unit.

[0076] According to the above configuration, vector information (optical flow) on the motion of the photographic subject captured in the first image can be derived (acquired) from the plurality of second images obtained by the second photographing unit.

[0077] In this case, by making the frame rate of the second imaging unit greater than the frame rate of the first imaging unit, the second imaging unit can output a plurality of second images corresponding to one first image output by the first imaging unit.

[0078] In addition, in the photographic system of the present invention,

[0079] The number of pixels of the second image output by the second imaging unit may be smaller than the number of pixels of the first image output by the first imaging unit.

[0080] According to the above configuration, since the signal volume output from the second imaging unit is suppressed, the readout time and signal processing time of the image sensor and the like in the second imaging unit are suppressed.

[0081] In addition, in the photographic system of the present invention,

[0082] The second photographing unit may output an image of a partial area in the photographed image as the second image, so that the number of pixels of the output second image is smaller than the number of pixels of the first image output by the first photographing unit.

[0083] In this way, in the second image obtained by the second imaging unit, only pixel signals of a necessary area are output to derive the optical flow for correcting the first image obtained by the first imaging unit, thereby reducing the amount of data of the output second image.

[0084] Furthermore, the second imaging unit may output, as the second image, an image in which pixel signals of a plurality of pixels in a captured image are temporally and spatially superimposed.

[0085] According to the above configuration, in each pixel of the second image, pixel signals from multiple pixels in a pixel array such as an image sensor are summed and averaged, thereby increasing the signal intensity. This ensures that the signal intensity (luminance information) in each pixel of the second image is sufficiently high relative to the noise (noise), thereby significantly improving the signal-to-noise ratio (SN ratio). Specifically, in situations such as photography in dimly lit environments, although the signal intensity (luminance information) in each pixel of the captured image of the second imaging unit is low, the SN ratio can be significantly improved by summing and averaging the multiple pixel signals and increasing the signal intensity (luminance information) in each pixel to produce the output image (second image). This allows the motion vector (optical flow) of the subject to be accurately derived, even when photographing in dimly lit environments.

[0086] In addition, in the photographic system of the present invention,

[0087] The plurality of photographing units include a first angle-of-view photographing unit with a predetermined angle of view and a second angle-of-view photographing unit with a larger angle of view than the first angle-of-view photographing unit.

[0088] The first viewing angle photographing unit can be used as the first photographing unit to take photos.

[0089] The second-angle-of-view photographing unit can perform photographing as the second photographing unit.

[0090] For example,

[0091] The plurality of photographing units include a telephoto photographing unit capable of performing telephoto photography, a wide-angle photographing unit capable of performing wide-angle photography, and an ultra-wide-angle photographing unit capable of performing ultra-wide-angle photography.

[0092] When the telephoto photography unit functions as the first photography unit for photography, the wide-angle photography unit functions as the second photography unit for photography.

[0093] or,

[0094] When the wide-angle photographing unit functions as the first photographing unit to perform photographing, the ultra-wide-angle photographing unit functions as the second photographing unit to perform photographing.

[0095] In addition, the photographic system of the present invention,

[0096] It has a calculation processing unit connected to each photographic unit.

[0097] The first photographing unit performs multiple exposures for dynamic range amplification photography, spatial resolution improvement based on multi-frame synthesis, or noise reduction based on multi-frame synthesis.

[0098] The arithmetic processing unit can use the optical flow to correct the jitter in each exposure and simultaneously correct the deviation between multiple exposures.

[0099] According to the above configuration, by correcting the shake in each exposure, the deviation between the multiple exposures is corrected at the same time, thereby obtaining a clearer photographic image.

[0100] In addition, the photographic system of the present invention,

[0101] It has a calculation processing unit connected to each photographic unit.

[0102] The first photographing unit performs multiple exposures for dynamic range amplification photography.

[0103] The arithmetic processing unit may use the second image in conjunction with saturation region determination during exposure of each of the first images.

[0104] According to the above configuration, the saturated area of ​​the first image is corrected, thereby obtaining a clear image.

[0105] The imaging device of the present invention includes any of the above-mentioned imaging systems.

[0106] Furthermore, the smartphone of the present invention includes any of the above-mentioned photography systems.

[0107] Furthermore, the photographing method of the present invention comprises:

[0108] Taking a first image of a photographic subject for external output, and taking a plurality of second images of the photographic subject for deriving an optical flow;

[0109] deriving an optical flow from the plurality of second images;

[0110] The first image is corrected using the derived optical flow.

[0111] According to the above configuration, even if jitter or defocus is caused by the movement of the photographed subject or the shaking of the smartphone in the first image obtained by photography, the jitter or defocus can be removed or suppressed by correcting the first image using optical flow, thereby obtaining a clear image.

[0112] The frame rate of the photography of the second image may be higher than the frame rate of the photography of the first image.

[0113] Hereinafter, one embodiment of the present invention will be described with reference to the accompanying drawings.

[0114] As shown in Figure 1 , the photographic system 1 of this embodiment includes multiple photographic units (cameras) 2 and a control unit 3 for controlling the multiple photographic units 2, enabling the capture of both still and moving images. This photographic system 1 can be incorporated into a digital camera, smartphone, tablet, or other imaging device. The photographic system 1 of this embodiment is incorporated into a smartphone 100. In other words, the photographic system 1 of this embodiment comprises the multiple photographic units (cameras) 2 and the control unit 3 of the smartphone 100.

[0115] Specifically, as shown in FIG2 , the smartphone 100 includes multiple imaging units (cameras) 2. Furthermore, the smartphone 100 includes a control unit 3 capable of controlling each imaging unit 2. Furthermore, the smartphone 100 includes a nonvolatile memory 101, a work memory 102, an operation unit 103, a display unit 104, a storage medium 105, a connection unit 106, a near-field wireless communication unit 107, a public network connection unit 108, a microphone 109, and a speaker 110. In the smartphone 100 of this embodiment, as described above, the multiple imaging units 2 and the control unit 3 constitute the imaging system 1. However, the control unit 3 may be built into any one of the imaging units 2, and the imaging system 1 may be composed of multiple imaging units 2.

[0116] Under the control of the control unit 3, each of the multiple imaging units 2 converts an image of a photographic subject (object) formed by an optical system 21 of the imaging unit 2 into an electrical signal, performs noise reduction processing, and outputs the digital data as an output image (image data). Specifically, each imaging unit 2 includes an optical system 21 composed of at least one optical element such as a lens, and a solid-state imaging element 22 that converts the image of the photographic subject formed by the optical system 21 into an electrical signal. The solid-state imaging element 22 is an element for photographing the photographic subject and has a pixel array composed of a plurality of pixels arranged in a matrix. The solid-state imaging element 22 in this embodiment is, for example, a CMOS image sensor.

[0117] The smartphone 100 of this embodiment includes three imaging units 2. The three imaging units 2 have different viewing angles and focal lengths. Specifically, the three imaging units 2 are a telephoto imaging unit 2A capable of telephoto photography, a wide-angle imaging unit 2B capable of wide-angle photography, and an ultra-wide-angle imaging unit 2C capable of ultra-wide-angle photography. For example, the focal length of the telephoto imaging unit 2A is greater than 80 mm in terms of 35 mm film, the focal length of the wide-angle imaging unit 2B is 24 mm in terms of 35 mm film, and the focal length of the ultra-wide-angle imaging unit 2C is less than 14 mm in terms of 35 mm film. Furthermore, the viewing angle (angle of field) of the wide-angle imaging unit 2B is greater than that of the telephoto imaging unit 2A, and the viewing angle of the ultra-wide-angle imaging unit 2C is greater than that of the wide-angle imaging unit 2B.

[0118] In addition, at least one of the three photographic units 2A, 2B, and 2C is configured to be able to switch between a first photographic mode and a second photographic mode. In the smartphone 100 of this embodiment, the wide-angle photographic unit 2B and the ultra-wide-angle photographic unit 2C are configured to be able to switch photographic modes, and the control unit 3 switches the photographic modes of the wide-angle photographic unit 2B and the ultra-wide-angle photographic unit 2C between the first photographic mode and the second photographic mode, respectively. The first photographic mode is a photographic mode for performing normal photography, and the second photographic mode is a photographic mode for deriving (acquiring) optical flow (motion vector of the photographic object) (a photographic mode for acquisition). In addition, the telephoto photographic unit 2A always photographs in the first photographic mode. Hereinafter, the photographic units 2A, 2B, and 2C whose photographic modes are the first photographic mode are sometimes referred to as the first photographic unit M1, and the photographic units 2B and 2C whose photographic modes are the second photographic mode are sometimes referred to as the second photographic unit M2.

[0119] In the photographic unit 2, the frame rate in the first photographic mode (when photographing with the first photographic unit M1) is different from the frame rate in the second photographic mode (when photographing with the second photographic unit M2). Specifically, the second photographic unit M2 outputs a plurality of corresponding image data (second images) for one image data (first image) output from the first photographic unit M1. That is, the frame rate of the second photographic unit M2 is greater than the frame rate of the first photographic unit M1. In this embodiment, the frame rate of the first photographic unit M1 is, for example, 60 fps, and the frame rate of the second photographic unit M2 is, for example, 1000 fps. In addition, the frame rates of the photographic units M1 and M2 are not limited to these values.

[0120] Furthermore, in the photographic unit 2, the number of pixels of the first image output in the first photographic mode (when photographing as the first photographic unit M1) differs from the number of pixels of the second image output in the second photographic mode (when photographing as the second photographic unit M2). Specifically, the number of pixels of the second image output by the second photographic unit M2 is smaller than the number of pixels of the first image output by the first photographic unit M1. This reduces the time required to read the photographic image from the second photographic unit M2. Furthermore, by reducing the size of the second image output from the second photographic unit M2, the amount of processing and the processing time required for the second image can be reduced by the control unit 3 and the later-described arithmetic processing unit 31.

[0121] Specifically, the second imaging unit M2 temporally and spatially overlaps the pixel signals of multiple pixels in the photographic image, thereby reducing the number of pixels (resolution) of the output second image to that of the first image output by the first imaging unit M1. For example, the second imaging unit M2 suppresses the number of pixels (resolution) by adding the pixel signals of each nearby pixel in the solid-state imaging element 22 (specifically, the pixel array) (overlapping the pixel signals of adjacent pixels temporally and spatially). This increases the SN ratio (signal-to-noise ratio) even when the brightness of the photographic subject is low. The overlapped (added, averaged, etc.) data can be digital data or analog data before DA conversion.

[0122] Furthermore, the second imaging unit M2 can output an image of a partial area Ar (see FIG. 4 ) of the captured image as the second image, thereby reducing the number of pixels in the output second image to that in the first image output by the first imaging unit M1. This area Ar corresponds to the image captured by the telephoto imaging unit 2A (the angle of view of the telephoto imaging unit 2A) when the wide-angle imaging unit 2B is acting as the second imaging unit M2, and also corresponds to the image captured by the wide-angle imaging unit 2B (the angle of view of the wide-angle imaging unit 2B) when the ultra-wide-angle imaging unit 2C is acting as the second imaging unit M2.

[0123] Specifically, in the smartphone 100 of this embodiment, the number of pixels (resolution) of the first image output from the first imaging unit M1 is, for example, 12.5 Mega pixels, and the number of pixels (resolution) of the second image output from the second imaging unit M2 is, for example, 750,000 pixels.

[0124] The control unit 3 controls each unit of the smartphone 100 based on input signals or programs. Furthermore, the control unit 3 generates a captured image (an image for external output) based on the signals (image data) output from each solid-state imaging element 22, and outputs it to the display unit 104. The control unit 3 of this embodiment also serves as the control unit of the imaging system 1. Furthermore, the control unit that controls each unit of the smartphone 100 is provided separately from the control unit 3 of the imaging system 1.

[0125] The control unit 3 controls one of the plurality of imaging units 2 as a first imaging unit M1 to perform imaging of a subject (normal imaging), while simultaneously controlling at least one of the remaining imaging units 2 as a second imaging unit M2 to perform acquisition imaging for deriving (acquiring) an optical flow of the subject. The control unit 3 includes a processing unit 31 directly or indirectly connected to each imaging unit 2. The processing unit 31 derives an optical flow from a second image (image data) output by the second imaging unit M2, and uses the derived optical flow to correct the first image (image data) output by the first imaging unit M1.

[0126] Here, optical flow refers to the optical motion of points between two temporally consecutive image frames in a digital image. This motion is observable by projecting the motion of an object, etc., onto the image. For example, optical flow vectorizes the differences between characteristic points (points exhibiting motion) between two temporally adjacent image frames in a dynamic image. In the imaging system 1 of this embodiment, each imaging unit 2B and 2C (specifically, the CMOS sensor of each imaging unit 2B and 2C) performs multiple imaging operations at a higher frame rate than normal imaging (i.e., performs imaging as the second imaging unit M2). From two temporally adjacent second image pairs among the multiple second images obtained through this imaging, the processing unit 31 calculates the vectors (movement direction and distance) of each point exhibiting motion, thereby deriving the optical flow.

[0127] Furthermore, the control unit 3 switches the wide-angle imaging unit 2B and the ultra-wide-angle imaging unit 2C to the first imaging mode or the second imaging mode, respectively.

[0128] Specifically, as shown in Figure 3, when an input is received from the operating unit 103 while the telephoto camera unit 2A is photographing a subject (normal photography), the control unit 3 controls the wide-angle camera unit 2B to switch to the second photography mode and perform photography as the second photography unit M2. Specifically, the control unit 3 causes the telephoto camera unit 2A to perform normal photography and output a first image, while also causing the wide-angle camera unit 2B to perform acquisition photography and output a second image. While the telephoto camera unit 2A does not switch photography modes, it performs photography as the first photography unit M1 for normal photography (photography in the first photography mode) and outputs the first image.

[0129] Furthermore, when an input is received from the operating unit 103 while the wide-angle camera unit 2B is photographing a subject (normal photography), the control unit 3 switches the wide-angle camera unit 2B to the first photography mode, causing it to perform photography as the first photography unit M1, and simultaneously switches the ultra-wide-angle camera unit 2C to the second photography mode, causing it to perform photography as the second photography unit M2. Specifically, the control unit 3 causes the wide-angle camera unit 2B to perform normal photography and output a first image, while causing the ultra-wide-angle camera unit 2C to perform acquisition photography and output a second image.

[0130] When the operation processing unit 31 derives (calculates) the optical flow from the second image output from the second photographic unit M2, it generates an event signal quantized as a three-value +1 / 0 / -1 based on the difference between the continuous images (second images), and after reducing the data transmission bandwidth, uses the generated event signal to derive the optical flow. Among them, +1 means that the brightness increases to a certain proportion or more, 0 means that the change in brightness is within a certain range, and -1 means that the brightness decreases to a certain degree or more. In addition, when the operation processing unit 31 generates an event signal based on the second image, it can generate event signals based on R, G, and B respectively, and can also generate event signals based on the brightness signal (Y). In addition, when there is sufficient data transmission bandwidth, the operation processing unit 31 can also directly calculate and derive the optical flow based on the brightness signal with grayscale. In addition, the method of deriving (calculating) the optical flow is not limited to the above method. The optical flow can also be derived using various existing methods (for example, the gradient method, the Lucas-Kanade method, etc.).

[0131] The nonvolatile memory 101 is an electrically erasable and storable nonvolatile memory. The nonvolatile memory 101 of this embodiment stores an OS (Operating System) as basic software executed by the control unit 3 and applications that realize application functions in cooperation with the OS.

[0132] The working memory 102 is used as a memory for image display of the display unit 104 , a working area of ​​the control unit 3 , and the like.

[0133] The operation unit 103 allows a user or the like to input instructions to the smartphone 100. The operation unit 103 of this embodiment includes a power button for turning the smartphone 100 on and off, a touch panel formed on the display unit 104, and the like.

[0134] The display unit 104 displays (outputs to the outside) a captured image (image data), displays characters for operation, and the like.

[0135] The storage medium 105 stores the image data output from the imaging unit 2 .

[0136] The connection unit 106 is an interface for connecting to an external device. Through the connection unit 106, the smartphone 100 and the external device exchange data.

[0137] The NFC unit 107 is a communication unit for performing NFC wireless communication and is composed of an antenna for wireless communication and a modulation and demodulation circuit or a communication controller for performing wireless signal processing.

[0138] Public network connection unit 108 is an interface for public wireless communications. Smartphone 100 can communicate with other devices through public network connection unit 108 for the purpose of making phone calls. Control unit 3 inputs and outputs audio signals via microphone 109 and speaker 110 to facilitate these calls. In this embodiment, public network connection unit 108 is an antenna, which connects control unit 3 to the public network.

[0139] In the smartphone 100 having the above-described configuration, blurring and defocusing of a captured image can be corrected as follows.

[0140] For example, when a photographer switches the imaging unit 2 used for photography to the wide-angle imaging unit 2B through input from the operating unit 103, the wide-angle imaging unit 2B is switched to the first imaging mode by the control unit 3, while the ultra-wide-angle imaging unit 2C is switched to the second imaging mode. Thus, the wide-angle imaging unit 2B performs photography as the first imaging unit M1, while the ultra-wide-angle imaging unit 2C performs photography as the second imaging unit M2.

[0141] Next, the photographic subject is photographed by the photographic unit 2 of the photographic system 1 (smartphone 100). In the photographic system 1 of this embodiment, for example, while the first photographic unit M1 (wide-angle photographic unit 2B) captures a photographic image (first image) at 60 frames per second (fps) / 12.5 mega pixels, the second photographic unit M2 (ultra-wide-angle photographic unit 2C) captures a photographic image (second image) for deriving (acquiring) optical flow at 1000 fps / 75000 pixels.

[0142] At this time, the control unit 3 adjusts the focus position of the second imaging unit M2 based on the focus information of the first imaging unit M1 , that is, makes the focuses of the two imaging units M1 and M2 consistent.

[0143] In addition, the peripheral portion of the photographic image obtained by the second photographic unit M2 (ultra-wide-angle photographic unit 2C) is unnecessary for deriving the optical flow of the photographic image (first image) output from the first photographic unit M1 (wide-angle photographic unit 2B). Therefore, the control unit 3 limits the readout area Ar of the photographic image obtained from the second photographic unit M2 in consideration of the viewing angle and parallax of the first photographic unit M1, thereby outputting the second image more quickly.

[0144] At this time, the control unit 3 limits the readout area Ar based on the focus information of the first imaging unit M1 (in this embodiment, the wide-angle imaging unit 2B). In addition, the limit of the readout area Ar is performed taking into account the large parallax when the focused imaging subject is close.

[0145] Alternatively, it is possible to fixedly cut out an area with the maximum parallax assumed from the outset, and extract and use a necessary area for appropriate processing.

[0146] When the second imaging unit M2 outputs multiple second images, the processing unit 31 derives optical flow and uses this derived optical flow to correct the corresponding first images. This corrects for jitter or defocus caused by the subject's motion in the first images, resulting in sharp images.

[0147] In addition, in the corrected first image, the control unit 3 of this embodiment determines the reliability of the photographic image corrected by optical flow, and outputs the photographic image without correction if the reliability is low (the first image output from the first photographing unit M1).

[0148] The reliability is determined by, for example, object recognition based on AI, etc. At this time, for example, when a face is not determined to be a face, it is determined that the reliability is low.

[0149] Furthermore, the second image output from the second imaging unit M2 can also be used in the reliability assessment. Specifically, the second image output from the second imaging unit M2 reflects information about low resolution and a poor SN ratio, but no jitter. Therefore, the reliability of the correction result is determined based on this second image. Furthermore, the reliability of the first image corrected using optical flow is assessed. If the reliability is low, correction is performed again using the second image output from the second imaging unit M2. Specifically, the second image output from the second imaging unit M2 reflects information about low resolution and a poor SN ratio, but no jitter. Therefore, this image information replaces the image information from the first imaging unit M1 or performs a fusion.

[0150] The photography system 1 of the smartphone 100 includes a plurality of photography units 2. The plurality of photography units 2 include a first photography unit M1 that outputs a first image for external output by photographing a subject, and a second photography unit M2 that outputs a second image from which optical flow is derived by photographing the subject.

[0151] According to the above configuration, the first imaging unit M1 captures a first image of the subject (an image obtained by normal photography), while the second imaging unit M2 captures a second image of the subject (an image used for optical flow derivation). Thus, even if the first image captured by the first imaging unit M1 is shaken or defocused due to movement of the subject or shaking of the smartphone, the optical flow correction of the first image can suppress such shake or defocus, resulting in a clear image.

[0152] Furthermore, when the first imaging unit M1 continuously captures images at a certain frame rate for capturing moving images, it is possible to generate moving image information at a frame rate exceeding the original frame rate of the first imaging unit M1 using optical flow.

[0153] In addition, when the first photographic unit M1 performs multiple exposures for dynamic range amplification photography, spatial resolution enhancement based on multi-frame synthesis (Multi-Frame Superresolution: MFNR), and noise reduction based on multi-frame synthesis (Multi-Frame Noise Reduction: MFNR), not only can the optical flow be used to perform shake correction for each exposure (the photographic image in each exposure), but also deviation correction can be performed between multiple exposures (between the photographic images in each exposure).

[0154] In addition, when the first photographic unit M1 performs dynamic range amplification photography based on multi-frame synthesis, since the second photographic unit M2 performs high frame rate photography, the characteristic that the signal is not easily saturated even for high-brightness subjects can be utilized, and the second image can be used in conjunction with the saturation area judgment in the respective exposures of the first image.

[0155] Furthermore, the imaging system 1 of this embodiment includes a processing unit 31 connected to each imaging unit 2. The processing unit 31 derives an optical flow from the second image output from the second imaging unit M2 and corrects the first image output from the first imaging unit M1 using the optical flow.

[0156] Furthermore, a photography method using the photography system 1 of the smartphone 100 includes capturing a first image of a photographic subject for external output and capturing multiple second images of the photographic subject for deriving optical flow; deriving optical flow from the multiple second images; and correcting the first image using the derived optical flow. Thus, even if the captured first image exhibits jitter or defocus due to, for example, movement of the photographic subject or shaking of the smartphone, the jitter or defocus can be removed or suppressed by correcting the first image using optical flow, resulting in a clear image.

[0157] With the above configuration, the photographic system 1 can correct the first image (photographed image) output from the first photographing unit M1 using the optical flow corresponding to the first image. Furthermore, in the smartphone 100 of this embodiment, optical flow can be used to correct both still and moving images.

[0158] In addition, in the photographic system 1 of this embodiment, at least one photographic unit 2B, 2C among the multiple photographic units 2 can be switched between a first photographic mode and a second photographic mode. The at least one photographic unit 2B, 2C in the first photographic mode can perform photographing as a first photographic unit M1, and the at least one photographic unit 2B, 2C in the second photographic mode can perform photographing as a second photographic unit M2.

[0159] According to the above configuration, by switching the photographing mode, at least one photographing unit 2 (the wide-angle photographing unit 2B and the ultra-wide-angle photographing unit 2C in this embodiment) can be used as the first photographing unit M1 and the second photographing unit M2.

[0160] Furthermore, in the imaging system 1 of this embodiment, the second imaging unit M2 outputs multiple second images corresponding to a single first image output by the first imaging unit M1. This configuration allows vector information (optical flow) regarding the motion of the subject captured in the first image to be derived (acquired) from the multiple second images obtained by the second imaging unit M2.

[0161] In the imaging system 1 of this embodiment, by making the frame rate of the second imaging unit M2 higher than the frame rate of the first imaging unit M1 , the second imaging unit M2 outputs a plurality of second images corresponding to one first image output from the first imaging unit M1 .

[0162] At this time, by suppressing the number of pixels of the second image output by the second imaging unit M2 to be smaller than the number of pixels of the first image output by the first imaging unit M1, the readout time and signal processing time from the image sensor in the second imaging unit M2 can be suppressed.

[0163] Furthermore, in the imaging system 1 of this embodiment, the second imaging unit M2 has a wider angle of view than the first imaging unit M1. Therefore, the second imaging unit M2 outputs an image of a portion of the captured image, region Ar, as the second image, thereby reducing the number of pixels in the output second image to that in the first image output by the first imaging unit M1. This reduces the amount of data in the output second image, etc., by outputting only the pixel signals of region Ar required for deriving the optical flow used to correct the first image obtained from the first imaging unit M1.

[0164] Furthermore, the second imaging unit M2 outputs an image in which pixel signals of a plurality of pixels are temporally and spatially overlapped (by adding or averaging pixel signals of adjacent pixels, etc.) as a second image in the captured image.

[0165] According to the above configuration, the signal level is increased in each pixel of the second image by adding and averaging the pixel signals of nearby pixels in a pixel array such as an image sensor. This ensures sufficient signal level (brightness information) for each pixel of the second image relative to noise (noise), thereby significantly improving the signal-to-noise ratio (SN / S ratio). In other words, even when photographing in dimly lit environments, while the signal level (brightness information) is low in each pixel of the image captured by the second photographing unit M2, the signal level (brightness information) in each pixel is increased by adding and averaging the multiple pixel signals and then generating the output image (second image), thereby significantly improving the SN / S ratio. Therefore, even when photographing in dimly lit environments, the motion vector (optical flow) of the photographed subject can be accurately derived.

[0166] In addition, in the photographic system 1 of the present embodiment, the multiple photographic units 2 include a first-angle photographic unit with a specified angle of view (in the example of the present embodiment, the telephoto photographic unit 2A and the wide-angle photographic unit 2B), and a second-angle photographic unit with a angle of view greater than that of the first-angle photographic unit 2A, 2B (in the example of the present embodiment, the wide-angle photographic unit 2B and the ultra-wide-angle photographic unit 2C). The first-angle photographic units 2A, 2B perform photography as the first photographic unit M1 (normal photography), and the second-angle photographic units 2B, 2C perform photography as the second photographic unit M2 (photography for deriving (acquiring) optical flow).

[0167] Specifically, the multiple photographic units 2 include a telephoto photographic unit 2A capable of performing telephoto photography, a wide-angle photographic unit 2B capable of performing wide-angle photography, and an ultra-wide-angle photographic unit 2C capable of performing ultra-wide-angle photography. When the telephoto photographic unit 2A performs photography as the first photographic unit M1, the wide-angle photographic unit 2B performs photography as the second photographic unit M2, or, when the wide-angle photographic unit 2B performs photography as the first photographic unit M1, the ultra-wide-angle photographic unit 2C performs photography as the second photographic unit M2.

[0168] Furthermore, the imaging system 1, the imaging device equipped with the imaging system 1, and the smartphone 100 equipped with the imaging system 1 of the present invention are not limited to the above-described embodiments. Various modifications are possible without departing from the scope of the present invention. For example, the structure of one embodiment may be supplemented with the structure of another embodiment, or a portion of the structure of one embodiment may be replaced with the structure of another embodiment. Furthermore, a portion of the structure of an embodiment may be deleted.

[0169] The photography system 1 of the above embodiment is provided in the smartphone 100, but the present invention is not limited to this structure. The photography system 1 may also be provided in other imaging devices such as a digital camera, a tablet device (mobile terminal), etc.

[0170] Furthermore, the above-described imaging system 1 includes three imaging units 2A, 2B, and 2C, but the present invention is not limited to this configuration. The imaging system 1 may include two imaging units 2 or four or more imaging units 2 .

[0171] In the imaging system 1 of the above embodiment, the angles of view and focal lengths of the plurality of imaging units 2 are different from one another, but the present invention is not limited to this configuration. The angles of view or focal lengths of the plurality of imaging units 2 may all be the same, or only some of the imaging units 2 may be the same.

[0172] Furthermore, in the imaging system 1 of the above embodiment, the plurality of imaging units 2 include imaging units 2B and 2C that can switch between a first imaging mode and a second imaging mode, but the present invention is not limited to this configuration. The plurality of imaging units 2 may include imaging units that function only as the first imaging unit M1 and imaging units that function only as the second imaging unit M2. In other words, among the plurality of imaging units 2, there may be fixed imaging units 2 that function as the first imaging unit M1 and imaging units 2 that function as the second imaging unit M2.

[0173] Furthermore, the imaging system 1 of the above embodiment includes a processing unit 31 that derives an optical flow from the second image (image data) output by the second imaging unit M2 and corrects the first image (image data) output by the first imaging unit M1 using the derived optical flow. However, the imaging system 1 is not limited to this configuration. The imaging system 1 may also be configured not to include the processing unit 31, but to utilize a processing unit 31 provided in another device connected to the imaging system 1 to correct the first image.

[0174] Furthermore, in the imaging system 1 of the above-described embodiment, when the imaging unit 2 for normal photography (the imaging unit 2 that outputs the first image by photographing) is switched to the telephoto imaging unit 2A or the wide-angle imaging unit 2B, the imaging unit 2 for optical flow extraction (the imaging unit 2 that outputs the second image by photographing) is automatically selected to be the wide-angle imaging unit 2B or the ultra-wide-angle imaging unit 2C. However, this configuration is not limiting. Alternatively, a configuration may be employed in which the photographer manually selects whether to perform photography for optical flow extraction. Alternatively, for example, a configuration may be employed in which the imaging unit 22 for normal photography automatically switches to photography for optical flow extraction when the accumulation time of incident light on the solid-state imaging element 22 is equal to or greater than 1 / 1000 second.

[0175] Furthermore, in the imaging system 1 of the above-described embodiment, of the two imaging units 2 with different viewing angles, the imaging unit 2 with the smaller viewing angle functions as the first imaging unit M1, while the imaging unit 2 with the larger viewing angle functions as the second imaging unit M2. However, this configuration is not limiting. Alternatively, the imaging unit 2 with the larger viewing angle functions as the first imaging unit M1, while the imaging unit 2 with the smaller viewing angle functions as the second imaging unit M2. With this configuration, optical flow can be derived within the range of the second image obtained from the imaging unit 2 with the smaller viewing angle, and correction using optical flow can be performed within the range corresponding to the second image (such as the area outside the peripheral portion) within the first image obtained from the imaging unit 2 with the larger viewing angle.

[0176] Furthermore, the photographing system 1 may be configured to perform video frame interpolation (Video Frame Interpolation) on the output of the first photographing unit M1 using an optical flow derived from a photographed image (second image) of the second photographing unit M2 .

[0177] Furthermore, in the imaging system 1 of the above embodiment, a single imaging unit 2 functions as the second imaging unit M2 for deriving optical flow, capturing an image (first image) captured by a single first imaging unit M1. However, this configuration is not limiting. In an imaging system 1 having three or more imaging units 2, multiple imaging units 2 may function as the second imaging units M2 for deriving optical flow, capturing an image (first image) captured by a single first imaging unit M1. This configuration allows for the combined use of multiple pieces of information (output from the second imaging unit M2) to achieve higher spatial and temporal optical flow accuracy.

[0178] Furthermore, when the ultra-wide-angle camera unit 2C is used to obtain optical flow (i.e., used as the second camera unit M2) and to correct a dynamic image captured by the wide-angle camera unit 2B (first camera unit M1), as shown in FIG5B , the entire image captured by the ultra-wide-angle camera unit 2C is not required; only the portion corresponding to the image (field of view) captured by the wide-angle camera unit 2B is required. FIG5A shows an image captured by the wide-angle camera unit 2B, and FIG5B shows an image captured by the ultra-wide-angle camera unit 2C.

[0179] In this case, block readout from the CMOS image sensor, etc., such as by skipping the first 500 rows and the last 500 rows out of a total of 3000 rows, and skipping the first 500 rows and the last 500 rows out of a total of 4000 rows, can reduce the number of rows and columns to be accessed, thereby reducing the amount of data read out from the ultra-wide-angle imaging unit 2C and increasing the frame rate used for image acquisition. This can increase the temporal resolution of optical flow acquisition, or, when the temporal resolution is fixed, by pausing the sensor during a pause between frames, achieve the same optical flow at lower power.

[0180] To illustrate the present invention, the present invention has been appropriately and fully described above through embodiments with reference to the accompanying drawings. However, it should be noted that those skilled in the art can easily modify and / or improve the above embodiments. Therefore, as long as the modifications or improvements implemented by those skilled in the art do not depart from the scope of the claims set forth in the claims, such modifications or improvements should be construed as being included within the scope of the claims.

[0181] Explanation of the reference numerals: 1…photographic system, 2…photographic unit, 2A…telephoto photographic unit, 2B…wide-angle photographic unit, 2C…ultra-wide-angle photographic unit, 21…optical system, 22…solid-state photographing element, 3…control unit, 31…arithmetic processing unit, 100…smartphone, 101…non-volatile memory, 102…working memory, 103…operation unit, 104…display unit, 105…storage medium, 106…connection unit, 107…near-field wireless communication unit, 108…public network connection unit, 109…microphone, 110…speaker, Ar…area corresponding to the photographic image of the first photographic unit, M1…first photographic unit, M2…second photographic unit.

Claims

1. A photographic system, wherein: Equipped with multiple photography departments, The plurality of photographing units include a first photographing unit that outputs a first image for external output by photographing a photographic object, and a second photographing unit that outputs a second image for deriving an optical flow by photographing the photographic object.

2. The photographic system according to claim 1, wherein: A calculation processing unit connected to each photographic unit is provided. The arithmetic processing unit derives the optical flow from the second image output from the second imaging unit, and corrects the first image output from the first imaging unit using the optical flow.

3. The photographic system according to claim 1 or 2, wherein: At least one of the plurality of photographing units is capable of switching between a first photographing mode and a second photographing mode, The at least one photographing unit in the first photographing mode performs photographing as the first photographing unit, and the at least one photographing unit in the second photographing mode performs photographing as the second photographing unit.

4. The imaging system according to any one of claims 1 to 3, wherein: The second imaging unit outputs a plurality of second images corresponding to one first image output from the first imaging unit.

5. The photographic system according to claim 4, wherein: The frame rate of the second imaging unit is greater than the frame rate of the first imaging unit.

6. The imaging system according to any one of claims 1 to 5, wherein: The number of pixels of the second image output by the second imaging unit is smaller than the number of pixels of the first image output by the first imaging unit.

7. The photographic system according to claim 6, wherein: The second photographing unit outputs an image of a partial region in the photographed image as the second image, thereby making the number of pixels of the output second image smaller than the number of pixels of the first image output by the first photographing unit.

8. The imaging system according to any one of claims 1 to 6, wherein: The second imaging unit outputs, as the second image, an image in which pixel signals of a plurality of pixels in a photographed image are temporally and spatially superimposed.

9. The imaging system according to any one of claims 1 to 7, wherein: The plurality of photographing units include a first angle-of-view photographing unit with a predetermined angle of view and a second angle-of-view photographing unit with an angle of view greater than that of the first angle-of-view photographing unit. The first viewing angle photographing unit performs photographing as the first photographing unit, The second-angle-of-view photographing unit performs photographing as the second photographing unit.

10. The imaging system according to any one of claims 1 to 8, wherein: The plurality of photographing units include a telephoto photographing unit capable of performing telephoto photography, a wide-angle photographing unit capable of performing wide-angle photography, and an ultra-wide-angle photographing unit capable of performing ultra-wide-angle photography. When the telephoto photography unit performs photography as the first photography unit, the wide-angle photography unit performs photography as the second photography unit, or, When the wide-angle photographing unit performs photographing as the first photographing unit, the super wide-angle photographing unit performs photographing as the second photographing unit.

11. The imaging system according to any one of claims 1 to 10, wherein: A calculation processing unit connected to each photographic unit is provided. The first photographing unit performs multiple exposure for dynamic range amplification photography, spatial resolution improvement based on multi-frame synthesis, or noise reduction based on multi-frame synthesis, The arithmetic processing unit uses the optical flow to correct jitter in each exposure and also corrects deviations between a plurality of exposures.

12. The imaging system according to any one of claims 1 to 10, wherein: A calculation processing unit connected to each photographic unit is provided. The first photographing unit performs multiple exposures for dynamic range amplification photography, The arithmetic processing unit uses the second image for saturated region determination in respective exposures in coordination with the first image.

13. An imaging device comprising the imaging system according to any one of claims 1 to 12.

14. A smart phone comprising the photography system according to any one of claims 1 to 12.

15. A photographic method comprising: photographing a first image of a photographic object for external output, and photographing a plurality of second images of the photographic object for deriving an optical flow; deriving an optical flow from the plurality of second images; The first image is corrected using the derived optical flow.

16. The photographing method according to claim 15, wherein: The frame rate of photographing the second image is higher than the frame rate of photographing the first image.

Images