A shooting system, an imaging device equipped with a shooting system, and a smartphone equipped with a shooting system.

Abstract

A photographing system comprising a plurality of photographing units, the plurality of photographing units including a first photographing unit that outputs a first image for external output by photographing a subject, and a second photographing unit that outputs a second image for deriving an optical flow by photographing the subject.

Description

Technical Field

[0001] The present invention relates to a photographing system, an imaging device including the photographing system, and a smartphone including the photographing system.

Background Art

[0002] Conventionally, an imaging device (such as a smartphone) capable of photographing a photographing object such as a person, an object, or a landscape has been known. This imaging device includes a photographing unit having an optical system such as a lens and a solid-state imaging device such as an image sensor (Japanese Patent Application Laid-Open No. 2023-1788).

[0003] When photographing a photographing object with this imaging device, blurring or defocusing may occur in the obtained photographed image due to the fast movement of the photographing object or the shake of the imaging device during photographing. In particular, when the frame rate is reduced (that is, when the exposure time during photographing is increased) due to the surrounding being dim, etc., the blurring or defocusing in the photographed image becomes prominent.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] Therefore, an object of the present invention is to provide a photographing system, an imaging device including the photographing system, and a smartphone including the photographing system that can clearly photograph a moving imaging object.

Means for Solving the Problems

[0006] The photographing system according to the present invention includes a plurality of photographing units, The plurality of imaging units include a first imaging unit that outputs a first image for external output by imaging the object to be photographed, and a second imaging unit that outputs a second image for deriving optical flow by imaging the object to be photographed.

[0007] This shooting system Each imaging unit is equipped with a processing unit connected to a calculation unit. The calculation processing unit may derive the optical flow from the second image output by the second imaging unit, and correct the first image output by the first imaging unit using the optical flow.

[0008] Furthermore, in the aforementioned shooting system, The control unit is configured such that at least one of the plurality of imaging units can be switched between a first imaging mode and a second imaging mode. In the first shooting mode, the at least one shooting unit may perform shooting as the first shooting unit, and in the second shooting mode, the at least one shooting unit may perform shooting as the second shooting unit.

[0009] Furthermore, in the aforementioned shooting system, The second imaging unit may output a number of corresponding second images for a single first image data output from the first imaging unit.

[0010] Furthermore, in the aforementioned shooting system, The frame rate of the second imaging unit may be higher than the frame rate of the first imaging unit.

[0011] Furthermore, in the aforementioned shooting system, The number of pixels in the second image output by the second imaging unit may be smaller than the number of pixels in the first image output by the first imaging unit.

[0012] Furthermore, in the aforementioned shooting system, The second imaging unit may output an image of a portion of the captured image as the second image, thereby reducing the number of pixels in the output second image to less than the number of pixels in the first image output by the first imaging unit.

[0013] Furthermore, in the aforementioned shooting system, The second imaging unit may output as the second image an image obtained by superimposing the pixel signals of multiple pixels in the captured image in time and space.

[0014] Furthermore, in the aforementioned shooting system, The plurality of imaging units include a first field-of-view imaging unit with a predetermined field of view, and a second field-of-view imaging unit with a field of view larger than that of the first field-of-view imaging unit. The aforementioned first angle-of-view shooting unit performs shooting as the aforementioned first shooting unit, The second angle-of-view shooting unit may perform shooting as the second shooting unit.

[0015] Furthermore, in the aforementioned shooting system, The aforementioned plurality of shooting units include a telephoto shooting unit capable of telephoto shooting, a wide-angle shooting unit capable of wide-angle shooting, and an ultra-wide-angle shooting unit capable of ultra-wide-angle shooting. When the telephoto shooting unit performs shooting as the first shooting unit, the wide-angle shooting unit performs shooting as the second shooting unit. Or, When the wide-angle shooting unit performs the shooting as the first shooting unit, the ultra-wide-angle shooting unit may perform the shooting as the second shooting unit.

[0016] Furthermore, in the aforementioned shooting system, Each imaging unit is equipped with a processing unit connected to a calculation unit. The first imaging unit performs multiple exposures for dynamic range expansion, spatial resolution improvement by combining multiple frames, or noise reduction by combining multiple frames. The arithmetic processing unit may use the optical flow to correct blur in each exposure, as well as correct the discrepancy between multiple exposures.

[0017] Furthermore, in the aforementioned shooting system, It includes an arithmetic processing unit connected to each imaging unit, The first imaging unit performs multiple exposures for dynamic range expansion imaging, The arithmetic processing unit may use the second image in accordance with the saturation region determination in each exposure of the first image.

[0018] In addition, the imaging device according to the present invention includes any one of the above imaging systems.

[0019] In addition, the smartphone according to the present invention includes any one of the above imaging systems.

[0020] In addition, the imaging method according to the present invention is capturing a first image for external output with respect to an imaging object and capturing a plurality of second images for deriving an optical flow with respect to the imaging object, deriving an optical flow from the plurality of second images, and correcting the first image using the derived optical flow.

[0021] In addition, in the imaging method, the frame rate of capturing the second image may be higher than the frame rate of capturing the first image.

Brief Description of the Drawings

[0022] [Figure 1] FIG. 1 shows a functional block diagram of a smartphone according to the present embodiment [Figure 2] FIG. 2 is a view of the smartphone seen from the imaging unit side. [Figure 3] FIG. 3 is a diagram for explaining the switching between the first imaging unit and the second imaging unit of each imaging unit. [Figure 4] FIG. 4 is a diagram for explaining the correction of a captured image using an optical flow in an imaging system included in the smartphone. [Figure 5A] FIG. 5A is a schematic diagram showing an example of a first image. [Figure 5B] Figure 5B is a schematic diagram illustrating block readout, showing the area in the second image where block readout is performed. [Modes for carrying out the invention]

[0023] The imaging system according to the present invention is Equipped with multiple camera units, The plurality of imaging units include a first imaging unit that outputs a first image for external output by imaging the object to be photographed, and a second imaging unit that outputs a second image for deriving optical flow by imaging the object to be photographed.

[0024] With this configuration, the first imaging unit captures a first image of the subject (an image obtained through normal imaging), and the second imaging unit captures a second image of the subject (an image used to derive the optical flow). As a result, even if blurring or out-of-focus images occur in the first image captured by the first imaging unit due to the movement of the subject or the shaking of the smartphone, the optical flow can be used to correct the first image, thereby eliminating or suppressing the blurring or out-of-focus images and obtaining a clear image.

[0025] Furthermore, if the first shooting unit is continuously shooting at a certain frame rate for video recording, it is possible to generate video information exceeding the original frame rate of the first shooting unit using optical flow.

[0026] Furthermore, the present invention's imaging system is Each imaging unit is equipped with a processing unit connected to a calculation unit. The calculation processing unit may derive the optical flow from the second image output by the second imaging unit, and correct the first image output by the first imaging unit using the optical flow.

[0027] With this configuration, the first image (captured image) output from the first imaging unit can be corrected using an optical flow corresponding to the first image in the imaging system.

[0028] Furthermore, in the imaging system of the present invention, At least one of the aforementioned multiple shooting units is switchable between a first shooting mode and a second shooting mode. In the first shooting mode, the at least one shooting unit may perform shooting as the first shooting unit, and in the second shooting mode, the at least one shooting unit may perform shooting as the second shooting unit.

[0029] With this configuration, at least one of the shooting units can be used as a first shooting unit and as a second shooting unit by switching the shooting mode.

[0030] Furthermore, in the imaging system of the present invention, The second imaging unit may output a number of second images corresponding to a single first image output from the first imaging unit.

[0031] With this configuration, vector information (optical flow) about the movement of the subject captured in the first image can be derived (acquired) from multiple second images obtained by the second imaging unit.

[0032] In this case, the frame rate of the second imaging unit may be set higher than the frame rate of the first imaging unit so that the second imaging unit outputs multiple second images corresponding to one first image output from the first imaging unit.

[0033] Furthermore, in the imaging system of the present invention, The number of pixels in the second image output by the second imaging unit may be smaller than the number of pixels in the first image output by the first imaging unit.

[0034] With this configuration, the capacity of the signal output from the second imaging unit is reduced, thereby reducing the readout time from the image sensor and other components in the second imaging unit, as well as the signal processing time.

[0035] Furthermore, in the imaging system 1 of the present invention, The second imaging unit may output an image of a portion of the captured image as the second image, thereby making the number of pixels in the output second image smaller than the number of pixels in the first image output by the first imaging unit.

[0036] In this way, by outputting only the pixel signals of the region necessary to derive the optical flow used for correcting the first image obtained from the first imaging unit, the amount of data in the output second image can be reduced.

[0037] Furthermore, the second imaging unit may output an image as the second image obtained by superimposing the pixel signals of multiple pixels in the captured image in time and space.

[0038] With this configuration, the signal quantity increases in each pixel of the second image because the pixel signals of multiple pixels in the pixel array of the image sensor, etc., are added and averaged. As a result, sufficient signal quantity (luminance information) is ensured in each pixel of the second image against noise, and the signal-to-noise ratio (SNR) can be significantly improved. In other words, when shooting in dimly lit environments, the signal quantity (luminance information) in each pixel of the image captured by the second imaging unit is low. However, by adding and averaging multiple pixel signals to increase the signal quantity (luminance information) in each pixel and then outputting it as the output image (second image), the SNR can be significantly improved. This makes it possible to accurately derive the motion vector (optical flow) of the object being photographed, even when shooting in dimly lit environments.

[0039] Furthermore, in the imaging system of the present invention, The plurality of imaging units include a first field-of-view imaging unit with a predetermined field of view, and a second field-of-view imaging unit with a field of view larger than that of the first field-of-view imaging unit. The aforementioned first angle-of-view shooting unit performs shooting as the aforementioned first shooting unit, The second angle-of-view shooting unit may perform shooting as the second shooting unit.

[0040] for example, The aforementioned plurality of shooting units include a telephoto shooting unit capable of telephoto shooting, a wide-angle shooting unit capable of wide-angle shooting, and an ultra-wide-angle shooting unit capable of ultra-wide-angle shooting. When the telephoto shooting unit performs shooting as the first shooting unit, the wide-angle shooting unit performs shooting as the second shooting unit. Or, When the wide-angle shooting unit performs shooting as the first shooting unit, the ultra-wide-angle shooting unit may perform shooting as the second shooting unit.

[0041] Furthermore, the present invention's imaging system is Each imaging unit is equipped with a processing unit connected to a calculation unit. The first imaging unit performs multiple exposures for dynamic range expansion, spatial resolution improvement by combining multiple frames, or noise reduction by combining multiple frames. The arithmetic processing unit may use the optical flow to correct blur in each exposure, as well as correct the discrepancy between multiple exposures.

[0042] With this configuration, blurring in each exposure is corrected, as well as the shift between multiple exposures, resulting in sharper captured images.

[0043] Furthermore, the present invention's imaging system is Each imaging unit is equipped with a processing unit connected to a calculation unit. The aforementioned first imaging unit performs multiple exposures for dynamic range expansion imaging. The arithmetic processing unit may use the second image in accordance with the saturation region determination for each exposure of the first image.

[0044] With this configuration, the saturated region of the first image is corrected, resulting in a sharp image.

[0045] The imaging device according to the present invention comprises any of the above-described imaging systems.

[0046] Furthermore, the smartphone according to the present invention is equipped with any of the above-described shooting systems.

[0047] Furthermore, the photographic method according to the present invention is The process involves capturing a first image for external output from the target object, and simultaneously capturing multiple second images from the same target object to derive optical flow. To derive optical flow from the aforementioned multiple second images, The method comprises correcting the first image using the derived optical flow.

[0048] With this configuration, even if blurring or out-of-focus images occur in the first image obtained through shooting due to the movement of the subject or the shaking of the smartphone, the blurring or out-of-focus images can be removed or suppressed by correcting the first image using optical flow, resulting in a clear image.

[0049] In this case, the frame rate for capturing the second image may be higher than the frame rate for capturing the first image.

[0050] One embodiment of the present invention will be described below with reference to the attached drawings.

[0051] As shown in Figure 1, the shooting system 1 of this embodiment comprises a plurality of shooting units (cameras) 2 and a control unit 3 that controls the plurality of shooting units 2, and is capable of capturing still images and videos. This shooting system 1 is installed in imaging devices such as digital cameras, smartphones, and tablet devices. The shooting system 1 of this embodiment is installed in a smartphone 100. In other words, the shooting system 1 of this embodiment is configured using the plurality of shooting units (cameras) 2 and control unit 3 of the smartphone 100.

[0052] Specifically, as shown in Figure 2, this smartphone 100 is equipped with multiple camera units 2. The smartphone 100 also includes a control unit 3 capable of controlling each camera unit 2. The smartphone 100 also includes a non-volatile memory 101, a working memory 102, an operation unit 103, a display unit 104, a recording medium 105, a connection unit 106, a short-range wireless communication unit 107, a public network connection unit 108, a microphone 109, and a speaker 110. In the smartphone 100 of this embodiment, as described above, the multiple camera units 2 and the control unit 3 constitute the shooting system 1, but the control unit 3 may be built into one of the camera units 2, and the shooting system 1 may be composed of multiple camera units 2.

[0053] Each of the multiple imaging units 2, under the control of the control unit 3, converts the image of the subject (object) formed by the optical system 21 of the imaging unit 2 into an electrical signal, then performs noise reduction processing and other operations, and outputs the digital data as an output image (image data). Specifically, each imaging unit 2 has an optical system 21 composed of at least one optical element such as a lens, and a solid-state image sensor 22 that converts the image of the subject formed through the optical system 21 into an electrical signal. This solid-state image sensor 22 is an element for imaging the subject and has a pixel array composed of multiple pixels arranged in a matrix. The solid-state image sensor 22 in this embodiment is, for example, a CMOS image sensor.

[0054] The smartphone 100 of this embodiment is equipped with three shooting units 2. These three shooting units 2 have different angles of view and focal lengths. Specifically, these three shooting units 2 are a telephoto shooting unit 2A capable of telephoto shooting, a wide-angle shooting unit 2B capable of wide-angle shooting, and an ultra-wide-angle shooting unit 2C capable of ultra-wide-angle shooting. For example, the focal length of the telephoto shooting unit 2A is 80mm or more in 35mm film equivalent, the focal length of the wide-angle shooting unit 2B is 24mm in 35mm film equivalent, and the focal length of the ultra-wide-angle shooting unit 2C is 14mm or less in 35mm film equivalent. In addition, the angle of view (field of view) of the wide-angle shooting unit 2B is larger than the angle of view of the telephoto shooting unit 2A, and the angle of view of the ultra-wide-angle shooting unit 2C is larger than the angle of view of the wide-angle shooting unit 2B.

[0055] Furthermore, at least one of the three shooting units 2A, 2B, and 2C is configured to be switchable between a first shooting mode and a second shooting mode. In the smartphone 100 of this embodiment, the wide-angle shooting unit 2B and the ultra-wide-angle shooting unit 2C are configured to be switchable between shooting modes, and the control unit 3 switches the shooting modes of the wide-angle shooting unit 2B and the ultra-wide-angle shooting unit 2C to the first shooting mode and the second shooting mode, respectively. Here, the first shooting mode is the shooting mode for normal shooting, and the second shooting mode is the shooting mode for deriving (acquiring) optical flow (motion vector of the object being photographed) (a mode for performing acquisition shooting). Note that the telephoto shooting unit 2A always shoots in the first shooting mode. Hereinafter, the shooting units 2A, 2B, and 2C in the first shooting mode may be referred to as the first shooting unit M1, and the shooting units 2B and 2C in the second shooting mode may be referred to as the second shooting unit M2.

[0056] In the shooting unit 2, the frame rate in the first shooting mode (when shooting as the first shooting unit M1) is different from the frame rate in the second shooting mode (when shooting as the second shooting unit M2). Specifically, the second shooting unit M2 outputs multiple corresponding image data (second images) for each image data (first image) output from the first shooting unit M1. That is, the frame rate of the second shooting unit M2 is higher than the frame rate of the first shooting unit M1. In this embodiment, the frame rate of the first shooting unit M1 is, for example, 60fps, and the frame rate of the second shooting unit M2 is, for example, 1000fps. Note that the frame rates of each shooting unit M1 and M2 are not limited to these values.

[0057] Furthermore, in the imaging unit 2, the number of pixels in the first image output in the first imaging mode (when imaging as the first imaging unit M1) is different from the number of pixels in the second image output in the second imaging mode (when imaging as the second imaging unit M2). Specifically, the number of pixels in the second image output by the second imaging unit M2 is smaller than the number of pixels in the first image output by the first imaging unit M1. This reduces the readout time of the captured image in the second imaging unit M2. In addition, since the capacity of the second image output from the second imaging unit M2 is reduced, the amount and time of processing of the second image in the control unit 3 and the arithmetic processing unit 31, which will be described later, can be reduced.

[0058] More specifically, the second imaging unit M2 reduces the number of pixels (resolution) of the output second image to that of the first image output by the first imaging unit M1 by superimposing the pixel signals of multiple pixels in the captured image in time and space. For example, the second imaging unit M2 reduces the number of pixels (resolution) by adding the pixel signals of each pixel in the solid-state image sensor 22 (more specifically, the pixel array) in its vicinity (superimposing the pixel signals of adjacent pixels in time and space). This makes it possible to increase the signal-to-noise ratio (SNR) even when the brightness of the subject being photographed is low. The data that is superimposed (added, averaged, etc.) at this time may be digital data or analog data before DA conversion.

[0059] Furthermore, the second imaging unit M2 outputs an image of a certain region Ar (see Figure 4) of the captured image as a second image, thereby reducing the number of pixels in the output second image to that of the first image output by the first imaging unit M1. When the wide-angle imaging unit 2B is imaging as the second imaging unit M2, this region Ar corresponds to the image captured by the telephoto imaging unit 2A when it is imaging as the first imaging unit M1 (the field of view of the telephoto imaging unit 2A), and when the ultra-wide-angle imaging unit 2C is imaging as the second imaging unit M2, this region corresponds to the image captured by the wide-angle imaging unit 2B when it is imaging as the first imaging unit M1 (the field of view of the wide-angle imaging unit 2B).

[0060] Specifically, in the smartphone 100 of this embodiment, the number of pixels (resolution) of the first image output from the first imaging unit M1 is, for example, 12.5 megapixels, and the number of pixels (resolution) of the second image output from the second imaging unit M2 is, for example, 750,000 pixels.

[0061] The control unit 3 controls each part of the smartphone 100 according to the input signals and programs. The control unit 3 also generates an captured image (an image for external output) from the signals (image data) output from each solid-state image sensor 22 and outputs it to the display unit 104. In this embodiment, the control unit 3 also serves as the control unit for the shooting system 1. It should be noted that the control unit for controlling each part of the smartphone 100 and the control unit 3 for the shooting system 1 may be provided separately.

[0062] The control unit 3 controls one of the multiple imaging units 2 to perform imaging of the target object (normal imaging) as the first imaging unit M1, and at least one of the remaining imaging units 2 to perform acquisition imaging to derive (acquire) the optical flow of the target object as the second imaging unit M2. The control unit 3 has an arithmetic processing unit 31 that is directly or indirectly connected to each imaging unit 2. This arithmetic processing unit 31 derives the optical flow from the second image (image data) output from the second imaging unit M2 and corrects the first image (image data) output from the first imaging unit M1 with the derived optical flow.

[0063] Here, optical flow refers to the optical movement of each point between two temporally consecutive image frames, which can be observed by projecting the movement of an object being photographed onto the image in a digital image. For example, it is the vectorization of the difference between feature points (moving points) between two temporally adjacent image frames in a video. In the shooting system 1 of this embodiment, each shooting unit 2B, 2C (more specifically, the CMOS sensors of each shooting unit 2B, 2C) performs multiple shots at a higher frame rate than normal shooting (i.e., shooting is performed as a second shooting unit M2), and the processing unit 31 calculates the vectors (direction of movement and distance) of each moving point from each of two temporally adjacent pairs of second images acquired in this shooting, thereby deriving the optical flow.

[0064] Furthermore, the control unit 3 switches the wide-angle shooting unit 2B and the ultra-wide-angle shooting unit 2C to either the first shooting mode or the second shooting mode, respectively.

[0065] Specifically, as shown in Figure 3, when the control unit 3 receives input from the operation unit 103 that the telephoto shooting unit 2A is taking a picture of the object to be photographed (normal shooting), it switches the wide-angle shooting unit 2B to the second shooting mode and has it take a picture as the second shooting unit M2. In other words, the control unit 3 has the telephoto shooting unit 2A perform normal shooting and output the first image, and also has the wide-angle shooting unit 2B perform acquisition shooting and output the second image. The telephoto shooting unit 2A does not switch shooting modes, but since it performs normal shooting (shooting in the first shooting mode), it takes a picture as the first shooting unit M1 and outputs the first image.

[0066] Furthermore, when the control unit 3 receives input from the operation unit 103 that the wide-angle shooting unit 2B is to perform normal shooting of the target object, it switches the wide-angle shooting unit 2B to the first shooting mode and has it shoot as the first shooting unit M1, and also switches the ultra-wide-angle shooting unit 2C to the second shooting mode and has it shoot as the second shooting unit M2. In other words, the control unit 3 has the wide-angle shooting unit 2B perform normal shooting and output the first image, and has the ultra-wide-angle shooting unit 2C perform acquisition shooting and output the second image.

[0067] When the arithmetic processing unit 31 derives (calculates) the optical flow from the second image output from the second imaging unit M2, it creates an event signal quantized into three values ​​of +1 / 0 / -1 from the difference between consecutive images (second image), reduces the data transmission bandwidth, and then derives the optical flow using the created event signal. Here, +1 means that the brightness increases by more than a certain percentage, 0 means that the change in brightness is within a certain range, and -1 means that the brightness decreases by more than a certain amount. Furthermore, when the arithmetic processing unit 31 creates an event signal from the second image, it may create an event signal from R, G, and B respectively, or it may create an event signal based on the brightness signal (Y). If there is sufficient data transmission bandwidth, the arithmetic processing unit 31 may also derive the optical flow by directly calculating it from a brightness signal with gradation. Moreover, the method of deriving (calculating) the optical flow is not limited to the method described above. Various conventional methods (for example, the gradient method or the Lucas-Kanade method) may be used to derive the optical flow.

[0068] The non-volatile memory 101 is an electrically erasable and recordable non-volatile memory. In this embodiment, the non-volatile memory 101 stores the OS (operating system), which is the basic software executed by the control unit 3, and applications that work in cooperation with this OS to realize advanced functions.

[0069] The working memory 102 is used as the image display memory for the display unit 104 and as the working area for the control unit 3, etc.

[0070] The operation unit 103 is used by the user to input instructions for the smartphone 100. In this embodiment, the operation unit 103 includes a power button for instructing the smartphone 100 to turn on / off, and a touch panel formed on the display unit 104.

[0071] The display unit 104 displays (outputs to an external source) the captured image (image data) and displays text for operation, etc.

[0072] The recording medium 105 records the image data output from the imaging unit 2.

[0073] The connection unit 106 is an interface for connecting to an external device. The smartphone 100 exchanges data with the external device via this connection unit 106.

[0074] The short-range wireless communication unit 107 is a communication unit for performing short-range wireless communication. The short-range wireless communication unit 107 consists of an antenna for wireless communication and a modulation / demodulation circuit and communication controller for processing wireless signals.

[0075] The public network connection unit 108 is an interface for performing public wireless communication. Through this public network connection unit 108, the smartphone 100 communicates with other devices for making calls. At this time, the control unit 3 realizes the call by inputting and outputting audio signals via the microphone 109 and speaker 110. In this embodiment, the public network connection unit 108 is an antenna, and the control unit 3 connects to the public network via this antenna.

[0076] With the smartphone 100 configured as described above, blur and out-of-focus images can be corrected in the following way.

[0077] When the photographer or other user switches the shooting unit 2 used for shooting to the wide-angle shooting unit 2B based on input from the operation unit 103, the control unit 3 switches the wide-angle shooting unit 2B to the first shooting mode and the ultra-wide-angle shooting unit 2C to the second shooting mode. As a result, the wide-angle shooting unit 2B shoots as the first shooting unit M1, and the ultra-wide-angle shooting unit 2C shoots as the second shooting unit M2.

[0078] Next, the shooting unit 2 of the shooting system 1 (smartphone 100) takes a picture of the target. In the shooting system 1 of this embodiment, for example, when the first shooting unit M1 (wide-angle shooting unit 2B) takes a picture (first image) at 60 frames per second (fps) / 12.5 mega pixels, the second shooting unit M2 (ultra-wide-angle shooting unit 2C) takes a picture (second image) at 1000 fps / 75000 pixels to derive (acquire) the optical flow.

[0079] At this time, the control unit 3 adjusts the focal position of the second imaging unit M2 based on the focus information of the first imaging unit M1, that is, it matches the focus of both imaging units M1 and M2.

[0080] Furthermore, since the peripheral portion of the captured image obtained from the second imaging unit M2 (ultra-wide-angle imaging unit 2C) is unnecessary for deriving the optical flow of the captured image (first image) output from the first imaging unit M1 (wide-angle imaging unit 2B), the control unit 3 limits the readout area Ar of the captured image obtained from the second imaging unit M2, taking into account the field of view and parallax of the first imaging unit M1, thereby enabling faster output of the second image.

[0081] At this time, the limiting of the readout area Ar by the control unit 3 is performed based on the focus information of the first imaging unit M1 (in this embodiment, the wide-angle imaging unit 2B). Furthermore, the limiting of the readout area Ar is performed taking into consideration the large parallax that occurs when the in-focus subject is close by.

[0082] Alternatively, one could initially assume parallax from infinity to very close distances, fix the maximum region to be extracted, and then process the image to extract the necessary regions as needed.

[0083] When the second imaging unit M2 outputs multiple second images, the arithmetic processing unit 31 derives an optical flow and uses this derived optical flow to correct the corresponding first image. This corrects blur and out-of-focus areas in the first image caused by the movement of the subject, resulting in a clear image.

[0084] Furthermore, in this corrected first image, the control unit 3 of this embodiment determines the reliability of the captured image corrected using optical flow, and if the reliability is low, it outputs the captured image without correction (the first image as output from the first imaging unit M1).

[0085] This confidence level is determined using, for example, object recognition by AI. In this case, for instance, a low confidence level is determined when a human face is not recognized as a human face.

[0086] Furthermore, the second image output from the second imaging unit M2 may be used to determine the reliability. Specifically, the second image output from the second imaging unit M2 has low resolution and a poor signal-to-noise ratio, but it contains unblurred information, so the reliability of the correction result is determined based on this second image. In addition, the reliability of the first image corrected using optical flow may be determined, and if the reliability is low, the correction may be performed again using the second image output from the second imaging unit M2. Specifically, the second image output from the second imaging unit M2 has low resolution and a poor signal-to-noise ratio, but it contains unblurred information, so the image information from the first imaging unit M1 may be replaced or blended based on this image information.

[0087] The above-described smartphone 100 shooting system 1 includes a plurality of shooting units 2. The plurality of shooting units 2 include a first shooting unit M1 that outputs a first image for external output by shooting the target object, and a second shooting unit M2 that outputs a second image for deriving optical flow by shooting the target object.

[0088] With this configuration, the first imaging unit M1 captures a first image of the subject (an image obtained through normal imaging), and the second imaging unit M2 captures a second image of the subject (an image used to derive the optical flow). As a result, even if blurring or out-of-focus images occur in the first image captured by the first imaging unit M1 due to the movement of the subject or the shaking of the smartphone, the blurring or out-of-focus images can be removed or suppressed by correcting the first image using the optical flow, resulting in a clear image.

[0089] Furthermore, if the first shooting unit M1 is continuously shooting at a certain frame rate for video recording, it is possible to generate video information exceeding the original frame rate of the first shooting unit M1 using optical flow.

[0090] Furthermore, when the first imaging unit M1 is performing multiple exposures for dynamic range expansion, spatial resolution improvement by combining multiple frames (Multi-Frame Superresolution: MFNR), and noise reduction by combining multiple frames (Multi-Frame Noise Reduction: MFNR), optical flow can be used to correct not only the blur of each exposure (image captured at each exposure) but also the shift between multiple exposures (images captured at each exposure).

[0091] Furthermore, when the first imaging unit M1 is performing dynamic range expansion imaging by combining multiple frames, the second imaging unit M2 is performing high-frame-rate imaging, and by utilizing its characteristic of being less prone to signal saturation even for high-brightness subjects, the second image can be used in accordance with the saturation region determination for each exposure of the first image.

[0092] Furthermore, the shooting method using the shooting system 1 of the smartphone 100 includes capturing a first image for external output with respect to the subject, capturing multiple second images for deriving optical flow with respect to the subject, deriving optical flow from the multiple second images, and correcting the first image using the derived optical flow. As a result, even if blurring or out-of-focus images occur in the first image obtained by shooting due to the movement of the subject or the shaking of the smartphone, the blurring or out-of-focus images can be removed or suppressed by correcting the first image using optical flow, and a clear image can be obtained as a result.

[0093] Furthermore, the imaging system 1 of this embodiment includes an arithmetic processing unit 31 connected to each imaging unit 2. The arithmetic processing unit 31 derives optical flow from the second image output by the second imaging unit M2 and corrects the first image output by the first imaging unit M1 using the optical flow.

[0094] With this configuration, the shooting system 1 can correct the first image (captured image) output from the first shooting unit M1 using optical flow corresponding to the first image. In this embodiment, the smartphone 100 can perform correction using optical flow for both still images and videos.

[0095] Furthermore, in the shooting system 1 of this embodiment, at least one of the multiple shooting units 2, 2B, 2C, can be switched between a first shooting mode and a second shooting mode. In the first shooting mode, the at least one shooting unit 2B, 2C performs shooting as the first shooting unit M1, and in the second shooting mode, the at least one shooting unit 2B, 2C performs shooting as the second shooting unit M2.

[0096] With this configuration, at least one shooting unit 2 (in this embodiment, a wide-angle shooting unit 2B and an ultra-wide-angle shooting unit 2C) can be used as a first shooting unit M1 and as a second shooting unit M2 by switching the shooting mode.

[0097] Furthermore, in the imaging system 1 of this embodiment, the second imaging unit M2 outputs a plurality of second images corresponding to a single first image output from the first imaging unit M1. With this configuration, vector information (optical flow) about the movement of the object being photographed in the first image can be derived (acquired) from the plurality of second images obtained by the second imaging unit M2.

[0098] In the imaging system 1 of this embodiment, by setting the frame rate of the second imaging unit M2 higher than the frame rate of the first imaging unit M1, the second imaging unit M2 outputs a number of corresponding second images for each first image output from the first imaging unit M1.

[0099] In this case, by making the number of pixels in the second image output by the second imaging unit M2 smaller than the number of pixels in the first image output by the first imaging unit M1, the signal capacity can be reduced, thereby reducing the readout time from the image sensor and the signal processing time in the second imaging unit M2.

[0100] Furthermore, in the imaging system 1 of this embodiment, the field of view of the second imaging unit M2 is larger than that of the first imaging unit M1. For this reason, the second imaging unit M2 outputs an image of a portion of region Ar in the captured image as a second image, thereby reducing the number of pixels in the output second image to less than the number of pixels in the first image output by the first imaging unit M1. In this way, by outputting only the pixel signals of region Ar, which is necessary to derive the optical flow used for correcting the first image obtained from the first imaging unit M1, from the second image obtained by the second imaging unit M2, the amount of data such as the output second image can be reduced.

[0101] Furthermore, the second imaging unit M2 outputs a second image by superimposing the pixel signals of multiple pixels in the captured image in terms of time and space (by adding, averaging, etc., the pixel signals of neighboring pixels).

[0102] With this configuration, the pixel signals of neighboring pixels in the pixel array of the image sensor, etc., are added and averaged in each pixel of the second image, increasing the signal amount. As a result, sufficient signal amount (luminance information) is ensured in each pixel of the second image against noise, and the signal-to-noise ratio (SNR) can be significantly improved. In other words, when shooting in dimly lit environments, the signal amount (luminance information) in each pixel of the image captured by the second shooting unit M2 is low. However, by adding and averaging multiple pixel signals to increase the signal amount (luminance information) in each pixel before outputting the image (second image), the SNR can be significantly improved. This makes it possible to accurately derive the motion vector (optical flow) of the object being photographed, even when shooting in dimly lit environments.

[0103] Furthermore, in the imaging system 1 of this embodiment, the multiple imaging units 2 include a first field-of-view imaging unit with a predetermined field of view (in this embodiment, a telephoto imaging unit 2A and a wide-angle imaging unit 2B) and a second field-of-view imaging unit with a field of view larger than that of the first field-of-view imaging units 2A and 2B (in this embodiment, a wide-angle imaging unit 2B and an ultra-wide-angle imaging unit 2C). The first field-of-view imaging units 2A and 2B perform imaging as the first imaging unit M1 (normal imaging), and the second field-of-view imaging units 2B and 2C perform imaging as the second imaging unit M2 (imaging for deriving (acquiring) optical flow).

[0104] Specifically, the multiple shooting units 2 include a telephoto shooting unit 2A capable of telephoto shooting, a wide-angle shooting unit 2B capable of wide-angle shooting, and an ultra-wide-angle shooting unit 2C capable of ultra-wide-angle shooting. When the telephoto shooting unit 2A is shooting as the first shooting unit M1, the wide-angle shooting unit 2B is shooting as the second shooting unit M2, or when the wide-angle shooting unit 2B is shooting as the first shooting unit M1, the ultra-wide-angle shooting unit 2C is shooting as the second shooting unit M2.

[0105] It should be noted that the imaging system 1, the imaging device equipped with the imaging system 1, and the smartphone 100 equipped with the imaging system 1 of the present invention are not limited to the above embodiments, and various modifications can be made without departing from the spirit of the present invention. For example, the configuration of one embodiment can be added to the configuration of another embodiment, and a part of the configuration of one embodiment can be replaced with the configuration of another embodiment. Furthermore, a part of the configuration of one embodiment can be deleted.

[0106] The shooting system 1 in the above embodiment is provided in a smartphone 100, but is not limited to this configuration. The shooting system 1 may also be provided in other imaging devices such as a digital camera or a tablet device (mobile terminal).

[0107] Furthermore, although the above-described imaging system 1 includes three imaging units 2A, 2B, and 2C, it is not limited to this configuration. The imaging system 1 may include two imaging units 2, or four or more imaging units 2.

[0108] Furthermore, in the above embodiment of the shooting system 1, the angles of view and focal lengths of the multiple shooting units 2 are different from each other, but the system is not limited to this configuration. The angles of view and focal lengths of all the shooting units 2 may be the same, or some of the shooting units 2 may have the same angles of view and focal lengths.

[0109] Furthermore, in the above embodiment of the imaging system 1, the plurality of imaging units 2 include imaging units 2B and 2C that can be switched between a first imaging mode and a second imaging mode, but the system is not limited to this configuration. The plurality of imaging units 2 may include an imaging unit that functions only as a first imaging unit M1 and an imaging unit that functions only as a second imaging unit M2. That is, in the plurality of imaging units 2, the imaging unit 2 that functions as a first imaging unit M1 and the imaging unit 2 that functions as a second imaging unit M2 may be fixed.

[0110] Furthermore, the imaging system 1 of the above embodiment includes a calculation processing unit 31 that derives an optical flow from the second image (image data) output from the second imaging unit M2 and corrects the first image (image data) output from the first imaging unit M1 using the derived optical flow, but is not limited to this configuration. The imaging system 1 may not include a calculation processing unit 31, and may be configured to correct the first image using a calculation processing unit 31 provided in another device that can be connected to the imaging system 1.

[0111] Furthermore, in the above embodiment of the imaging system 1, when the imaging unit 2 used for normal imaging (imaging unit 2 that outputs the first image through imaging) is switched to the telephoto imaging unit 2A or the wide-angle imaging unit 2B, the wide-angle imaging unit 2B or the ultra-wide-angle imaging unit 2C is automatically selected as the imaging unit 2 for deriving optical flow (imaging unit 2 that outputs the second image through imaging), but the system is not limited to this configuration. The system may also be configured in which the photographer or the like manually selects whether or not to perform imaging for deriving optical flow. Alternatively, for example, the system may be configured in which it automatically switches to performing imaging for deriving optical flow when the accumulation time of incident light in the solid-state image sensor 22 for normal imaging is 1 / 1000 second or greater.

[0112] Furthermore, in the above embodiment of the imaging system 1, the imaging unit 2 with the smaller field of view is configured to perform imaging as the first imaging unit M1, and the imaging unit 2 with the larger field of view is configured to perform imaging as the second imaging unit M2, but the system is not limited to this configuration. The imaging unit 2 with the larger field of view may perform imaging as the first imaging unit M1, and the imaging unit 2 with the smaller field of view may perform imaging as the second imaging unit M2. With this configuration as well, optical flow can be derived in the range of the second image obtained from the imaging unit 2 with the smaller field of view, and correction using optical flow can be performed in the range of the first image obtained from the imaging unit 2 with the larger field of view that corresponds to the second image (such as the area excluding the periphery).

[0113] Furthermore, the imaging system 1 may be configured to perform video frame interpolation on the output of the first imaging unit M1 using the optical flow derived from the image (second image) captured by the second imaging unit M2.

[0114] Furthermore, in the above embodiment of the imaging system 1, there is one imaging unit 2 that captures an image (first image) from one first imaging unit M1 as a second imaging unit M2 for deriving optical flow, but the system is not limited to this configuration. In an imaging system 1 equipped with three or more imaging units 2, there may be multiple imaging units 2 that capture an image (first image) from one first imaging unit M1 as a second imaging unit M2 for deriving optical flow. With such a configuration, by combining and using multiple pieces of information (output from the second imaging unit M2), it is possible to obtain optical flow with higher spatial and temporal accuracy.

[0115] Furthermore, when using the ultra-wide-angle shooting unit 2C for optical flow acquisition (i.e., as the second shooting unit M2) and applying correction to the video shot by the wide-angle shooting unit 2B (first shooting unit M1), as shown in Figure 5B, not the entire image captured by the ultra-wide-angle shooting unit 2C is needed; only the portion corresponding to the image (field of view) captured by the wide-angle shooting unit 2B is required. Here, Figure 5A shows the image captured by the wide-angle shooting unit 2B, and Figure 5B shows the image captured by the ultra-wide-angle shooting unit 2C.

[0116] In such cases, block readout from a CMOS image sensor, for example, in the example shown in Figure 5B, by skipping the first 500 rows and the last 500 rows out of a total of 3000 rows, and the first 500 columns and the last 500 columns out of a total of 4000 columns, the number of rows and columns accessed can be reduced, thereby reducing the amount of data read from the ultra-wide-angle imaging unit 2C and increasing the frame rate for image acquisition. As a result, in optical flow acquisition, the temporal resolution of the optical flow can be increased, or if the temporal resolution is kept constant, a pause period can be introduced between frames to stop the sensor, thereby obtaining the same optical flow with lower power consumption.

[0117] In order to express the present invention, the embodiments have been adequately and sufficiently described above with reference to the drawings. However, those skilled in the art should recognize that it is easy to modify and / or improve upon the above embodiments. Therefore, unless such modifications or improvements implemented by those skilled in the art deviate from the scope of the claims, such modifications or improvements shall be considered to be included within the scope of the claims. [Explanation of Symbols]

[0118] 1...Shooting system, 2...Shooting unit, 2A...Telephoto shooting unit, 2B...Wide-angle shooting unit, 2C...Ultra-wide-angle shooting unit, 21...Optical system, 22...Solid-state image sensor, 3...Control unit, 31...Calculation processing unit, 100...Smartphone, 101...Non-volatile memory, 102...Working memory, 103...Operation unit, 104...Display unit, 105...Recording medium, 106...Connection unit, 107...Short-range wireless communication unit, 108...Public network connection unit, 109...Microphone, 110...Speaker, Ar...Area corresponding to the image captured by the first shooting unit, M1...First shooting unit, M2...Second shooting unit

Claims

1. It comprises multiple imaging units and a processing unit connected to each imaging unit, The plurality of imaging units include a first imaging unit that outputs a first image for external output by imaging the object to be photographed, and a second imaging unit that outputs a second image for deriving optical flow by imaging the object to be photographed. The calculation processing unit derives the optical flow from the second image output by the second imaging unit, and corrects the first image output by the first imaging unit using the optical flow. The first imaging unit performs multiple exposures for dynamic range expansion, spatial resolution improvement by combining multiple frames, or noise reduction by combining multiple frames, and the processing unit uses the optical flow to correct blur in each exposure and correct the shift between multiple exposures, thus forming an imaging system.

2. At least one of the aforementioned multiple shooting units is switchable between a first shooting mode and a second shooting mode. The shooting system according to claim 1, wherein the at least one shooting unit performs shooting as the first shooting unit in the first shooting mode, and the at least one shooting unit performs shooting as the second shooting unit in the second shooting mode.

3. The imaging system according to claim 1, wherein the second imaging unit outputs a plurality of second images corresponding to a single first image output from the first imaging unit.

4. The imaging system according to claim 3, wherein the frame rate of the second imaging unit is higher than the frame rate of the first imaging unit.

5. The imaging system according to claim 1, wherein the number of pixels of the second image output by the second imaging unit is smaller than the number of pixels of the first image output by the first imaging unit.

6. The imaging system according to claim 5, wherein the second imaging unit outputs an image of a portion of the captured image as the second image, thereby reducing the number of pixels in the output second image to less than the number of pixels in the first image output by the first imaging unit.

7. The imaging system according to claim 1, wherein the second imaging unit outputs an image as the second image obtained by superimposing the pixel signals of multiple pixels in the captured image in time and space.

8. The plurality of imaging units include a first field-of-view imaging unit with a predetermined field of view, and a second field-of-view imaging unit with a field of view larger than that of the first field-of-view imaging unit. The aforementioned first angle-of-view shooting unit performs shooting as the aforementioned first shooting unit, The shooting system according to claim 1, wherein the second angle of view shooting unit performs shooting as the second shooting unit.

9. The aforementioned plurality of shooting units include a telephoto shooting unit capable of telephoto shooting, a wide-angle shooting unit capable of wide-angle shooting, and an ultra-wide-angle shooting unit capable of ultra-wide-angle shooting. When the telephoto shooting unit performs shooting as the first shooting unit, the wide-angle shooting unit performs shooting as the second shooting unit. Or, The shooting system according to claim 1, wherein when the wide-angle shooting unit performs shooting as the first shooting unit, the ultra-wide-angle shooting unit performs shooting as the second shooting unit.

10. An imaging device comprising the imaging system described in any one of claims 1 to 9.

11. A smartphone comprising the shooting system described in any one of claims 1 to 9.

12. The process involves capturing a first image for external output from the target object, and simultaneously capturing multiple second images from the same target object to derive optical flow. To derive optical flow from the aforementioned multiple second images, The method comprises correcting the first image using the derived optical flow, A shooting method further comprising performing multiple exposures for dynamic range expansion, spatial resolution improvement by combining multiple frames, or noise reduction by combining multiple frames, and using the optical flow to correct blur in each exposure as well as the shift between multiple exposures.

13. The shooting method according to claim 12, wherein the frame rate for shooting the second image is higher than the frame rate for shooting the first image.

Images