Imaging device and its settings screen

Abstract

To improve usability during shooting or playback in an imaging device equipped with a plurality of imaging units. [Solution] In an imaging device equipped with multiple imaging units, a setting means for setting multiple pre-settable shooting modes is provided, and a means for controlling the multiple imaging units according to the set shooting mode and an operation means are also provided. This allows a user to select a desired shooting mode, allowing the multiple imaging units to be linked together to take images with a simple operation, and also makes it possible to control the multiple imaging units independently or simultaneously to take images, thereby providing a user-friendly imaging device.

Description

Technical Field

[0001] The present invention relates to a technique for controlling a plurality of imaging units in an imaging apparatus including the plurality of imaging units to perform imaging by causing each of the plurality of imaging units to operate in a different operation mode.

Background Art

[0002] An imaging apparatus such as a digital camera that captures an image of a subject using an imaging element such as a CCD or a CMOS sensor generally has a single imaging unit configured from a single imaging element and an optical system. In contrast, imaging apparatuses including a plurality of imaging units have also been proposed. As background art in this technical field, there is Japanese Patent Application Laid-Open No. 10-134187 (Patent Document 1). Patent Document 1 discloses an apparatus that includes a camera with a wide-angle field of view and a camera with a narrow-angle field of view, and is capable of estimating depth information with high accuracy.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of Invention

Problems to be Solved by the Invention

[0004] The apparatus of Patent Document 1 can achieve a wide-angle field of view and highly accurate depth estimation. However, no consideration was given to usability during shooting or playback.

Means for Solving the Problems

[0005] To achieve the above objectives, the present invention provides, as an example, an imaging device comprising a first imaging unit, a second imaging unit, a display unit, an image signal processing circuit, a control circuit, an operation unit, a mode setting unit, and a recording unit, wherein the first and second imaging units are arranged to image in the same direction with a predetermined distance between them, the image signal processing circuit generates a first image signal from the signal captured by the first imaging unit, a second image signal from the signal captured by the second imaging unit, and a third signal from the signal captured by the first and second imaging units, and the control circuit controls the first and second imaging units, the image signal processing circuit, and the recording unit with a single imaging operation by the operation unit, according to a plurality of shooting modes that can be set in advance by input from the mode setting unit, and the plurality of shooting modes are configured to include a shooting mode that records a first image signal, a mode that records a first image signal and a second image signal, and a mode that records a third signal. [Effects of the Invention]

[0006] According to the present invention, it becomes possible to control multiple imaging units simultaneously and take images with simple operation, thereby providing an easy-to-use imaging device. [Brief explanation of the drawing]

[0007] [Figure 1] This is a block diagram showing the configuration of the imaging device according to Example 1. [Figure 2] This is an external view showing an example of the imaging device in Example 1. [Figure 3] This is an external view showing another example of the imaging device in Example 1. [Figure 4] This is a list of shooting modes in Example 1. [Figure 5] This is an explanatory diagram showing the timing of the operation of normal shooting 2 in Example 1. [Figure 6] This is an explanatory diagram showing the imaging timing of the imaging unit in the high-speed continuous shooting mode according to Embodiment 1. [Figure 7] This is a list of shooting modes in Example 2. [Figure 8]FIG. 2 is a block diagram illustrating an example configuration of an image signal processing circuit in Embodiment 2. [Figure 9] FIG. 10 is an example of an image in Embodiment 2. [Figure 10] FIG. 11 is a diagram showing a distance histogram for the image in FIG. 9. [Figure 11] FIG. 12 is an example of a synthesized output image in Embodiment 2. [Figure 12] FIG. 13 is an example of a screen display of a user interface in a distance adaptive processing mode in Embodiment 2. [Figure 13] FIG. 14 is another example of a screen display of a user interface in a distance adaptive processing mode in Embodiment 2. [Figure 14] FIG. 15 is a list of shooting modes in Embodiment 3. [Figure 15] FIG. 16 is an example of a menu screen when individually performing settings for a first imaging unit and a second imaging unit in Embodiment 3. [Figure 16] FIG. 17 is an example of screen display during monitoring in Embodiments 1 to 3. [Figure 17] FIG. 18 is an explanatory diagram of an imaging form in Embodiment 4. [Figure 18] FIG. 19 is an external view illustrating an example of an imaging apparatus in Embodiment 4. [Figure 19] FIG. 20 is an external view illustrating another example of an imaging apparatus in Embodiment 4. [Figure 20] FIG. 21 is an external view illustrating an example of an imaging apparatus in Embodiment 5. [Figure 21] FIG. 22 is a block diagram illustrating a configuration of an imaging apparatus in Embodiment 6. [Figure 22] FIG. 23 is a list of normal shooting modes in Embodiment 6. [Figure 23] FIG. 24 is a list of simultaneous shooting modes in Embodiment 6. [Figure 24] FIG. 25 is an explanatory diagram of a relationship between a focal length and a zoom magnification in Embodiment 6. [Figure 25] FIG. 26 is an explanatory diagram of a screen display in a still image shooting mode in Embodiment 6. [Figure 26] FIG. 27 is an explanatory diagram of a screen display in a moving image shooting mode in Embodiment 6. [Figure 27] FIG. 4 is an explanatory diagram of a display screen during wide-angle image recording in a moving image shooting mode in Embodiment 6. [Figure 28] FIG. 7 is an explanatory diagram of a display screen during narrow-angle image recording in a moving image shooting mode in Embodiment 6. [Figure 29] FIG. 13 is an explanatory diagram of screen display in a simultaneous still image shooting mode in Embodiment 6. [Figure 30] FIG. 19 is an explanatory diagram of two-screen display in the simultaneous still image shooting mode in Embodiment 6. [Figure 31] FIG. 25 is an explanatory diagram of screen display in a simultaneous moving image shooting mode in Embodiment 6. [Figure 32] FIG. 31 is an explanatory diagram of screen display during simultaneous moving image recording in the simultaneous moving image shooting mode in Embodiment 6. [Figure 33] FIG. 37 is an explanatory diagram of screen display in a simultaneous moving image and still image shooting mode in Embodiment 6. [Figure 34] FIG. 43 is an explanatory diagram of screen display in a playback mode in Embodiment 6. [Figure 35] FIG. 49 is an explanatory diagram of another screen display in the playback mode in Embodiment 6. [Figure 36] FIG. 55 is an external view of an imaging apparatus in Embodiment 7. [Figure 37] FIG. 61 is an external view of an imaging apparatus in Embodiment 8. [Figure 38] FIG. 67 is another external view of the imaging apparatus in Embodiment 8. [Figure 39] FIG. 73 is an explanatory diagram of shooting mode screen display in Embodiment 8. [Mode for Carrying Out the Invention]

[0008] Hereinafter, embodiments of the present invention will be described with reference to the drawings. [Embodiment]

[0009] Figure 1 is a block diagram showing the configuration of the imaging device according to this embodiment. The imaging device 1 of this embodiment has two imaging units. 2a and 2b are lens units, and 3a and 3b are imaging units. The imaging units 3a and 3b are arranged to image in the same direction with a predetermined distance between them. The lens units 2a and 2b are composed of multiple lenses, including a focusing lens, and the control circuit 6 controls the position of the focusing lens to perform focusing.

[0010] The imaging units 3a and 3b are composed of image sensors such as CMOS or CCD. Photoelectric conversion elements are arranged in two dimensions on the imaging surfaces of the imaging units 3a and 3b, and each receives input via the lens units 2a and 2b, and converts the optical image of the subject formed on the imaging surface into an imaging signal through photoelectric conversion. The imaging units 3a and 3b have built-in AD conversion circuits that convert analog signals into digital signals and output the digitized imaging signal. Furthermore, the imaging units 3a and 3b may use image sensors with pixels for phase-detection AF (autofocus) to achieve faster autofocus speeds. The imaging units 3a and 3b may also have built-in memory to speed up the input and output of the captured signals. If an image sensor without a built-in AD conversion circuit is used, the AD conversion circuit may be provided outside the imaging units 3a and 3b. The imaging units 3a and 3b and the lens units 2a and 2b are controlled by the control circuit 6 according to the operating mode.

[0011] The interface circuit 4 outputs input signals from the imaging units 3a and 3b to the image signal processing circuit 5 according to the operating mode of the imaging device. In this process, depending on the operating mode, it outputs one or both of the imaging signals captured by the imaging units 3a and 3b. This operation is controlled by the control circuit 6.

[0012] The image signal processing circuit 5 performs various signal processing on the output signal from the interface circuit 4, including filtering, amplification according to the sensitivity setting, and white balance correction. The image signal processing circuit 5 also generates video data and still image data, which are video signals for display or video signals for recording, depending on the operating mode. These video signals for display and video signals for recording include video signals corresponding to the imaging units 3a and 3b, respectively, and video signals generated by combining the output signals of the imaging units 3a and 3b.

[0013] The image signal processing circuit 5 is connected to the control circuit 6, display 8, memory 12 such as DRAM or flash memory, encoding / decoding circuit 15, recording / playback circuit 10, etc., enabling high-speed coordinated processing with each part. The hardware configuration of the image signal processing circuit 5 and the control circuit 6 may be a single-chip LSI, or they may be configured as separate LSIs. In addition, the image signal processing circuit 5 and the recording / playback circuit 10 may be configured to have two signal processing and recording circuits corresponding to two imaging units, depending on the performance requirements of the imaging device, so that two signal processing systems can be performed simultaneously.

[0014] The control circuit 6 is connected to the lens units 2a and 2b, imaging units 3a and 3b, interface circuit 4, image signal processing circuit 5, attitude detection unit 7, operation input unit 9, position information input unit 13, wireless communication unit 14, etc. The control circuit 6 controls each of these connected units according to the operating mode. The control circuit 6 also controls the display 8, encoding / decoding circuit 15, and recording / playback circuit 10 either via the image signal processing circuit 5 or directly. The attitude detection unit 7 includes a gyro sensor and an electronic compass and can detect the shooting direction (azimuth angle, elevation angle) and image tilt of the imaging device. In this embodiment, the control circuit 6 controls the encoding and decoding circuit 15 to control the imaging units 3a and 3b, the image signal processing circuit 5, the display 8, the recording and playback circuit 10, etc., according to the operation mode set in advance by user operation, and controls the system to display or record one of the following in a single shooting operation: the image signal captured by the imaging unit 3a, the image signal captured by the imaging unit 3b, or the image signal generated by combining the image signals captured by the imaging unit 3a and the imaging unit 3b.

[0015] The encoding and decoding circuit 15 performs encoding processing for recording on the signal-processed video signal and also performs decoding processing on the reproduced image data. The recording and playback circuit 10 records the captured image data onto the recording medium 11 and also plays back the images recorded on the recording medium 11.

[0016] The operation input unit 9 accepts user input. Specifically, as will be described later, the mode setting dial, shutter button, mode setting button, etc., correspond to the operation input unit 9. The mode setting dial should be assigned to the most frequently used operations by the user among the settings for the imaging device 1.

[0017] Display 8 is a display that integrates, for example, a touch panel and a display unit such as a liquid crystal panel, and may also be provided with an optical or electronic viewfinder in addition to Display 8. The display unit is not limited to liquid crystal, and any other type such as organic EL may be used. Furthermore, the touch panel may use, for example, a capacitive type or a pressure-sensitive type.

[0018] Figure 2 is an external view showing an example of an imaging device in this embodiment, where (A) is a rear view, (B) is a right side view, and (C) is a front view. In Figure 2, the imaging device 1 has a display 8 with an integrated touch panel located on its rear side.

[0019] The lens units 2a and 2b are arranged at a predetermined interval so that they each have approximately the same optical axis. The lens units 2a and 2b may be fixed or detachable. The operation input section 9 of the imaging device 1 includes a mode setting dial 31, a shutter button 32, operation buttons 33, and a video recording button 34. 35 is a power switch. The mode setting dial 31 is used to select the type of shooting mode of the imaging device 1. The shutter button 32 is located on the upper right side of the main body, and pressing it is used to take a picture. The operation buttons 33 have a central button and buttons for the up, down, left, and right directions, and are used to display and select menu screens. An electronic viewfinder may also be provided on the top side of the imaging device 1.

[0020] The imaging device 1 has at least two operating modes: a shooting mode and a playback mode, and the shooting mode has multiple modes. These operating modes can be switched using the mode setting dial 31. A dedicated playback mode button may be provided for the playback mode. The types of shooting modes and their details will be described later, but here we will explain the overview of how the shooting modes work. When the control circuit 6 detects that a predetermined shooting mode has been set by the operation input unit 9, it performs control according to the shooting mode on the imaging units 3a and 3b, the image signal processing circuit 5, the display 8, the encoding / decoding circuit 15, the recording / playback circuit 10, etc. In each shooting mode, both still image and video recording are possible.

[0021] The control circuit 6 reads the imaging signal from the imaging unit 3a or 3b at a predetermined period, processes it via the interface circuit 4, performs predetermined signal processing in the image signal processing circuit 5, converts it to a display format, and displays the moving image on the display 8 in real time. The user monitors this moving image and takes pictures using the following actions.

[0022] When taking a still image, the user presses the shutter button 32. The control circuit 6 detects that the shutter button 32 has been pressed from the output of the operation input unit 9, and sets the aperture value, shutter speed, etc., according to the shooting conditions set by the user. The shutter button 32 may be one that has two states, half-press and fully-press, and may be controlled to determine the settings such as focus and shutter speed when it is half-pressed. These settings are performed by the control circuit 6 controlling the lens units 2a, 2b and the image sensors 3a, 3b. When the shutter button is operated, the camera controls which of the image sensors 3a, 3b to use for imaging, or both of the image sensors 3a, 3b to use, what kind of signal processing to perform, and which signals to record, according to a preset shooting mode. The imaging signal of a still image captured by the image sensors 3a, 3b is subjected to predetermined still image signal processing by the image signal processing circuit 5, and then encoded for still images by the encoding / decoding circuit 15. Memory 12 is used as a buffer memory when performing still image signal processing, encoding, etc. The encoding method is, for example, the JPEG format, but other methods such as MPEG or high-resolution RAW format may also be used for recording. The encoded still image data is recorded on the recording medium 11 by the recording and playback circuit 10 via the memory 12.

[0023] When recording a video, press the video recording button 34 to start recording, and press it again to stop recording. The video can be encoded in MPEG such as H264 or H265, or any other video format. In the case of video recording, the encoding / decoding circuit 15 encodes the video in a predetermined format, and the encoded video data is recorded on the recording medium 11 by the recording / playback circuit 10 via the memory 12.

[0024] When the control circuit 6 detects that the system has been set to playback mode based on input from the operation input unit 9, it reads still images and videos recorded on the recording medium 11 by the recording and playback circuit 10. Then, after decoding by the encoding and decoding circuit 15, it controls the system to display them on the display 8.

[0025] As described above, the imaging device according to this embodiment has multiple imaging units and enables the multiple imaging units to take images in conjunction with a single shooting operation such as pressing a shutter button, according to a preset operating mode.

[0026] Figure 3 is an external view showing another example of the imaging device 1 in this embodiment, where (A) is a rear view and (B) is a front view. In the example of Figure 3, the shape is vertically elongated compared to the example of Figure 2, and the display 8' is relatively larger than in the example of Figure 2(A). Also, in the example of Figure 3, the mode setting dial 31, shutter button 32, operation button 33, and video recording button 34 shown in Figure 2 are not provided, but icons corresponding to these operation buttons are displayed on the display 8', and the desired input can be made by touch panel operation.

[0027] Next, the shooting modes in this embodiment will be described. Figure 4 is a list of the shooting modes in this embodiment. There are five shooting modes: normal shooting 1, normal shooting 2, simultaneous shooting, high-speed continuous shooting, and video / still image shooting. Figure 4 shows the operation of the two imaging units, the recorded image, and an overview of the control in each mode. The first imaging unit corresponds to imaging unit 3a in Figure 1, and the second imaging unit corresponds to imaging unit 3b. "Enabled" means that the imaging operation is performed, and "Disabled" means that the imaging operation is not performed. Regarding the recorded image, "First" or "Second" means that the image captured by the first imaging unit or the second imaging unit is recorded, and "First and Second" means that both the images captured by the first and second imaging units are recorded.

[0028] Normal shooting 1 is a shooting mode that operates similarly to a camera with a single image sensor, but with only the first image sensor enabled. When capturing still images, the first image sensor performs both monitoring and still image capture, and when capturing video, the first image sensor also performs both monitoring and video capture. Since this mode is the same as any known imaging device with a single image sensor, a detailed explanation is omitted. Here, the first image sensor is "enabled," but it is also possible to "disable" the first image sensor and "enable" the second image sensor. In this shooting mode, only one image sensor is operated, which is advantageous for reducing power consumption.

[0029] Normal shooting 2 enables both the first and second imaging units, but the first imaging unit is used for monitoring and the second imaging unit is used for still image capture. Figure 5 is an explanatory diagram showing the timing of the operation of normal shooting 2 in this embodiment, with the horizontal axis representing time. In this mode, the first imaging unit reads out frame images at a predetermined period to generate a moving image for monitoring. Exposure is performed in parallel with the readout at the frame period. The readout moving image is converted into a display image by the image signal processing circuit 5 in Figure 1 and displayed on the display 8. The user monitors this moving image and presses the shutter button at the desired timing (shutter on in Figure 5). The second imaging unit starts exposure after the shutter on is detected by the control circuit 6, and reads out the captured still image signal when the exposure time corresponding to the set shutter speed is completed. The time required for readout is determined by the number of pixels in the still image and the readout speed, but generally it takes longer than the frame of the monitoring video in the first imaging unit.

[0030] In imaging devices with a single imaging unit, still image capture and pre-image monitoring are performed by the same imaging unit. As a result, monitoring is interrupted during still image capture, potentially causing the user to miss the next photo opportunity. With the operation of this embodiment, continuous monitoring is possible even during still image capture, making it less likely to miss a photo opportunity.

[0031] Next, let's explain the simultaneous shooting mode. In simultaneous shooting mode, the first and second image units capture images simultaneously. In this mode, the first image unit monitors and captures still images, while the second image unit captures still images at the same timing as the first image unit. In this mode, it is possible to change the shooting conditions (e.g., shutter speed, aperture, ISO sensitivity, effects, etc.) of the first and second image units and capture images simultaneously.

[0032] Next, the high-speed continuous shooting mode will be explained. Figure 6 is an explanatory diagram showing the imaging timing of the first and second imaging units in the high-speed continuous shooting mode of this embodiment. In this mode, the first and second imaging units take images alternately. If T is the time of one imaging cycle when taking still images at the fastest speed with a single imaging unit, then the number of still images that can be taken per second is 1 / T. In this case, it is assumed that the processing capacity required for signal processing and recording is sufficient for the imaging cycle. In the high-speed continuous shooting mode, by shifting the imaging timing of the first and second imaging units by T / 2, high-speed continuous shooting is possible at approximately twice the speed compared to when there is a single imaging unit.

[0033] Next, we will explain the video-still image shooting mode. In this mode, the first image unit captures video, and the second image unit captures still images. Because different image units are used for video and still image capture, it is possible to capture still images at any point during video recording. Unlike when capturing with a single image unit, there is no risk of part of the video being cut off when a still image is taken, or of the size of the still images recorded during video recording being restricted. In this mode, still image capture is performed when the shutter button 32 is pressed, and video recording is started and stopped by pressing the shutter button 32.

[0034] In this embodiment, by operating the shutter button 32, imaging with multiple imaging units is possible with the same simple operation as operating an imaging device with a single imaging unit, thus providing an easy-to-use imaging device. In addition, there are advantages such as the monitor not being interrupted during still image shooting, reducing the chance of missing a photo opportunity, and enabling high-speed shooting.

[0035] In this embodiment, five shooting modes were described as specific examples: normal shooting 1, normal shooting 2, simultaneous shooting, high-speed continuous shooting, and video / still image shooting. However, the system may have all five of these modes, or only some of them.

[0036] Thus, in this embodiment, an imaging device equipped with multiple imaging units is provided with setting means for setting multiple preset shooting modes, and means for controlling the multiple imaging units according to the set shooting mode, as well as operating means. This makes it possible to synchronize the imaging of multiple imaging units in response to a single shooting command from the operating means.

[0037] In other words, an imaging device comprising a first imaging unit, a second imaging unit, a display unit, an image signal processing circuit, a control circuit, an operation unit, a mode setting unit, and a recording unit, wherein the first and second imaging units are arranged to image in the same direction with a predetermined distance between them, the image signal processing circuit generates a first image signal from the signal captured by the first imaging unit, generates a second image signal from the signal captured by the second imaging unit, and generates a third signal from the signal captured by the first and second imaging units, the control circuit controls the first and second imaging units, the image signal processing circuit, and the recording unit with a single imaging operation by the operation unit, according to a plurality of shooting modes that can be set in advance by input from the mode setting unit, and the plurality of shooting modes are configured to include a shooting mode that records a first image signal, a mode that records a first image signal and a second image signal, and a mode that records a third signal.

[0038] This makes it possible to control and capture images from multiple imaging units simultaneously with simple operation, providing a user-friendly imaging device. [Examples]

[0039] Other embodiments of the present invention will now be described. The configuration of the imaging device in this embodiment is the same as in Embodiment 1 shown in Figure 1. The differences from Embodiment 1, namely the contents of the shooting modes, will be explained using Figure 7. Figure 7 is a list of the shooting modes in this embodiment. There are five shooting modes in this embodiment: distance measurement, background blur, distance adaptation, HDR, and 3D.

[0040] In distance measurement mode, the first imaging unit captures still images or videos, and the first and second imaging units calculate distance information for each part of the captured image. The calculated distance information is recorded together with the image information captured by the first imaging unit. The distance information is calculated as distance information for each pixel or for each region consisting of multiple adjacent pixels, using the parallax information of the first and second imaging units and known methods such as stereo matching. The distance information calculated in this way is recorded in the image file together with the captured image data, or as a separate file. The file format of the 3D information file recorded as a separate file is arbitrary, but it may be converted to polygon data such as the STL format used for 3D printing and recorded.

[0041] In this distance measurement mode, distance information for any point on the image can be obtained during shooting or playback. For example, an interface may be provided that displays the distance from the shooting position to the subject, or to any point on the screen, when the user selects a point on the touch panel.

[0042] In this distance measurement mode, the images to be recorded are those captured by the first imaging unit. However, it is also possible to record only the images captured by the second imaging unit instead of the images captured by the first imaging unit, or to record both the first and second images captured by the imaging units.

[0043] In this distance measurement mode, images captured by the first or second imaging unit are recorded, and distance information corresponding to each pixel or pixel area is calculated and recorded from the two images captured by the first and second imaging units. In this way, distance information can be obtained for any point in the captured image. As for the distance measurement method, an active distance measurement method may be used in combination, which uses a laser light emitter and a light receiver to measure the distance based on the time difference between emission and reception, thereby improving the distance measurement accuracy, especially for long distances.

[0044] Next, we will explain the background blur mode. In this mode, similar to the distance measurement mode explained earlier, distance information is calculated using images captured by the first and second imaging units. This distance information is then used to blur the background. We will now explain the details of this process.

[0045] Figure 8 is a block diagram showing an example of the configuration of the image signal processing circuit 5 in Figure 1. In Figure 8, 100 is the still image / video signal processing circuit, 101 is the distance information generation circuit, 102 is the distance histogram generation circuit, 103 is the long-distance image processing circuit, 104 is the short-distance image processing circuit, 105 is the image histogram generation circuit, 106 is the synthesis circuit, and 107 is the display signal generation circuit. The still image / video signal processing circuit 100 generates still image data and video data for recording from the still image data and video data output by the first or second imaging unit. The still image / video signal processing circuit 100 performs signal processing while appropriately buffering image data in the memory 12 (Figure 1) according to the control of the control circuit 6 (Figure 1). The still image / video signal processing circuit 100 may also incorporate a signal processing unit for subject recognition, etc., to recognize various subjects or perform person recognition processing.

[0046] The distance information generation circuit 101 generates distance information from the captured image of the first imaging unit and the captured image of the second imaging unit, either pixel by pixel or in units of pixel groups consisting of multiple adjacent pixels. When generating distance information, images captured at the same timing by the first imaging unit and the second imaging unit are used. In the following description, the frame image generated at a predetermined timing in the first imaging unit is called the first image, and the image generated at the same timing as the first image in the second imaging unit is called the second image. Distance information can be generated from the first image and the second image using a known method. For example, it can be obtained from the first image and the second image using the principle of triangulation. The distance information generation circuit 101 has an input unit that receives the first image data input from the first imaging unit and the second image data input from the second imaging unit as input signals. The distance information generation circuit 101 also has an input / output interface that allows reading and writing of the input first and second image data to and from the memory 12 (Figure 1). This makes it possible to perform various processes on any pixel in the first and second images. Here, pattern matching is used to determine which pixel in the second image corresponds to each pixel in the first image. The first and second imaging units are arranged parallel to the optical axis at a fixed distance B apart, and any pixel p1 in the first image is assumed to contain a parallax of δ, where p2 is the corresponding pixel in the second image determined by pattern matching. The distance information generation circuit 101 generates this δ for each pixel. At this time, the distance Z between the imaging unit and the subject can be calculated using the principle of triangulation, from the relationship Z = B × f ÷ δ. Here, f is the focal length of the imaging unit, and the focal lengths of the first and second imaging units are assumed to be equal. By calculating the distance image for each pixel in this way, distance information for any pixel in the first image can be obtained.

[0047] The distance histogram generation circuit 102 calculates a distance histogram, that is, information about the number of pixels at a given distance, or in other words, the frequency of pixels at a given distance. Figure 9 shows an example image, and Figure 10 shows the distance histogram for this image. This image includes a person 111 and a distant building 112 as subjects. In the distance histogram of Figure 10, 111' corresponds to the person and is at a close distance. 112' corresponds to the building and is at a far distance.

[0048] The still image / video signal processing circuit 100 has a separation means for separating image data into multiple signal groups belonging to a predetermined subject distance, and the subsequent long-distance image processing circuit 103 and short-distance image processing circuit 104 perform different signal processing according to the distance.

[0049] The long-range image processing circuit 103 performs signal processing on a portion of the image located at a distance. In this background blur mode, the long-range image processing circuit 103 applies a strong low-pass filter to the long-range image corresponding to the background. The image histogram generation circuit 105 calculates, for example, the number of pixels in the image that have a given brightness, or the frequency of those pixels. A histogram for the RGB signal may also be calculated and displayed on a screen.

[0050] The near-field image processing circuit 104 performs signal processing on the portion of the image that is at close range. In this background blur mode, the near-field image processing circuit 104 applies arbitrary signal processing suitable for the subject to the near-field image, such as a person. For example, in the distance histogram of Figure 10, the far-field image processing circuit 103 applies a strong low-pass filter to the far-field image, which represents the background.

[0051] Images processed with different signal processing depending on the distance by the near-range image processing circuit 104 and the far-range image processing circuit 103 are combined by the synthesis circuit 106, and a display signal generation circuit 107 generates a display image to be shown on the display 8 (Figure 1). Figure 11 shows an example of the output image after synthesis. 130 is the synthesized image displayed on the display, and 131 is the part corresponding to the person. 132 is the part corresponding to the background such as buildings, and is blurred by filter processing because it corresponds to the background.

[0052] The synthesis circuit 106 is also connected to the outputs of the distance histogram generation circuit 102 and the image histogram generation circuit 105, allowing not only the display of the image but also the display of the distance histogram and image histogram individually or superimposed on the image. Such a display can be achieved by displaying a menu that allows the user to selectively switch the content displayed on the screen, and configuring the system to allow the user to select. The image histogram and distance histogram may be displayed separately, or they may be integrated into a two-dimensional histogram display of distance and brightness by, for example, using the horizontal axis as distance and the vertical axis as brightness and representing the number of pixels by color.

[0053] The intensity and type of signal processing, such as the strength of the low-pass filter in the long-range image processing circuit 103 and the short-range image processing circuit 104, as well as the distance on the histogram at which these signal processing operations are switched, may be adaptively changed based on the shape of the histogram calculated by the distance histogram generation circuit 102. Furthermore, the processing may be combined with person recognition processing within the still image / video signal processing, changing depending on whether a person is detected or not.

[0054] In Figure 8, the long-distance image processing circuit 103 and the short-distance image processing circuit 104 are configured to perform different signal processing for two distance levels. However, it is also possible to divide the processing into three or more levels and perform different processing for each level depending on the distance.

[0055] By using this background blur mode, you can make foreground subjects such as people stand out while blurring the background, and even when using a small image sensor, you can produce images with a blurred background that look as if they were taken with a single-lens reflex camera.

[0056] Next, we will explain the distance-adaptive processing mode. While the background blur mode described earlier makes foreground subjects such as people stand out and blurs the background, the distance-adaptive processing mode is a more generalized version of background blur. Depending on the user's preference, it allows for different processing depending on the distance to the subject, including processing similar to background blur, and processing such as blurring the foreground. For this reason, an interface is provided that allows the user to change the processing settings.

[0057] Figure 12 shows an example of the user interface screen display in distance adaptive processing mode. For example, a menu is provided that allows the user to display the screen shown in Figure 12 from a menu screen. In Figure 12, 8' is the display screen of the display, and the display screen has three areas: a video display area 140, a distance histogram display area 143, and a specific distance image display area 145. The input image is the same image as the image in Figure 9 described earlier. The image displayed in the video display area 140 is the image after signal processing. The distance histogram display area 143 displays the distance histogram, as in the example shown in Figure 10, and simultaneously displays a selection frame 144. The position of this selection frame 144 can be changed by the user using the touch panel. The image corresponding to the position on the distance histogram area selected here is displayed in the distance histogram display area 143. In the distance histogram display area 143, the near-field portion corresponding to the person is selected using the selection frame 144, so the specific distance image display area 145 displays the input image 141' of the person. 146 is the image quality adjustment display, and here, as an example of image quality adjustment items, it displays a menu for adjusting the strength of the low-pass filter used for blurring. The user can adjust the strength of the low-pass filter using the touch panel, and can check the effect of the image quality adjustment in the video display area 140. In addition to the low-pass filter, menus for other image processing such as color correction may also be displayed.

[0058] Figure 13 shows an example of the screen display when the background area including the building is selected in the selection frame 144', whereas Figure 12 shows an example of the screen display when the person portion is selected in the distance histogram display area 143. Therefore, the input image 142' including the background area including the building is displayed in the image display area 145 at a specific distance.

[0059] In this distance-adaptive processing mode, the user can apply their preferred processing to any region that can be separated according to the distance to the subject in the image. In addition to background blurring, it is possible to correct the subject's color or blur the foreground. In this mode as well, similar to background blurring, the processing may be performed in multiple stages rather than just two stages depending on the distance. Furthermore, the user interface may display the image before and after processing side by side to make it easier to compare the effects of the image processing. It may also be possible to select the distance without displaying a distance histogram.

[0060] In Figure 7, HDR (High Dynamic Range) is a mode that combines images captured under different imaging conditions by the first and second imaging units to produce an image with less black crushing and white clipping. Here, "imaging conditions" does not refer to the shooting environment, but rather to the settings of the imaging device related to shooting, such as shutter speed, aperture, and ISO sensitivity.

[0061] Generally, longer exposure times result in images with a good signal-to-noise ratio (S / N), but they are prone to saturation and have poor reproduction of high-brightness areas. Conversely, shorter exposure times generally result in images with good reproduction of high-brightness areas, but the S / N ratio in low-brightness areas tends to deteriorate. Therefore, in this HDR mode, images with different exposure times are captured by the first and second image units. For example, the first image unit can be set to a high shutter speed and the second image unit to a low shutter speed, or the aperture value of the first image unit can be increased and the aperture value of the second image unit to a low value, or the ISO sensitivity of the first image unit to a high and ISO sensitivity of the second image unit to a low ISO sensitivity, or by changing combinations of these settings to capture images with a large dynamic range and images with a small dynamic range but good S / N ratio from the first and second image units respectively, and then combine them. When combining the images, the system is designed to produce an image with a wide dynamic range that effectively balances brightness, color reproduction, and S / N ratio in both low-brightness and high-brightness areas, according to the image histogram.

[0062] In an imaging device having only a single imaging unit, it is possible to continuously capture and combine images with varying exposure times. However, if the subject is moving, changes occur in the continuous images with different exposure times, often resulting in double images and failing to achieve the effect of expanding the dynamic range. In contrast, in this embodiment, images can be captured simultaneously by the first and second imaging units, thus expanding the dynamic range and reducing the likelihood of double images occurring.

[0063] Although this embodiment was described using Figure 7, the imaging device may be equipped with all of these modes, or it may be an imaging device that has some of these modes. Furthermore, these may also be modes that are combinations of modes. For example, it is possible to combine background blur and HDR. By successively capturing high-speed shutter images and low-speed shutter images with the first imaging unit and combining them, a known dynamic range expansion process can be performed, while background blur can be performed using distance information obtained from the first and second imaging units. In addition, the modes described in Figure 4 in Embodiment 1 may be combined as appropriate.

[0064] Next, the 3D mode is a mode in which the first imaging unit records an image for the right eye and the second imaging unit records an image for the left eye. When taking or playing back images taken in this 3D mode, the right eye image from the first imaging unit and the left eye image from the second imaging unit may be displayed side by side on the display 8, and a dedicated viewer may be attached to the display to prevent interference between the left and right images so that the left eye can see only the left eye image and the right eye can see only the right eye image, allowing the user to view the 3D image.

[0065] As described above, this embodiment, like Embodiment 1, enables multiple imaging units to take images in conjunction with a single shooting command from the operating means, and allows for simultaneous control of multiple imaging units with simple operation to take images in multiple shooting modes, thus providing a user-friendly imaging device. [Examples]

[0066] Other embodiments of the present invention will now be described. The configuration of the imaging device in this embodiment is the same as in Embodiment 1 shown in Figure 1. The differences from Embodiments 1 and 2, namely the shooting modes, will be explained using Figure 14. There are five shooting modes in this embodiment: program AE, aperture priority, shutter priority, dual, and manual.

[0067] The Program AE mode is a shooting mode in which the first and second image units each take pictures according to two different program diagrams. The program diagrams used in this mode are arbitrary. The user may be able to select two from multiple program diagrams. In this mode, the focus mode and timer settings for the first and second image units may be set individually. For example, the first image unit can be set to manual focus and the second image unit to autofocus, or the first image unit can be set to no timer and the second image unit to a timer, allowing the first image unit to take a picture by pressing the shutter button, and then the second image unit to take a picture after a predetermined time set by the timer.

[0068] Aperture-priority mode is a shooting mode in which the first and second image sensors each shoot according to two different aperture-priority AE settings. It allows shooting at two different aperture values ​​simultaneously.

[0069] Shutter-priority mode is a shooting mode in which the first and second image sensors each shoot according to two different shutter-priority AE settings. It allows simultaneous shooting at two different shutter speeds.

[0070] Dual Mode is a shooting mode in which the first and second image units shoot according to two different operating modes, each from Program AE, Aperture Priority, Shutter Priority, and Manual. For example, you can set the first image unit to Aperture Priority and the second image unit to Shutter Priority and shoot simultaneously. You can also shoot simultaneously using any combination of two modes selected from Program AE, Aperture Priority, Shutter Priority, and Manual.

[0071] Manual mode allows simultaneous shooting using two different manual settings for the first and second image sensors. The configurable shooting conditions are arbitrary, including ISO sensitivity, aperture value, shutter speed, as well as focus, timer, image quality, and special effects.

[0072] According to this embodiment, it is possible to take photos in two different shooting modes with varying shooting conditions, thereby increasing the frequency with which desired photos can be taken.

[0073] In this embodiment, Figure 15 shows an example of a menu screen for individually configuring the settings of the first and second imaging units. Items 800 to 804 are setting items for the first imaging unit, and in order are the setting menus for ISO sensitivity, aperture value, shutter speed, timer, and flash. For example, the desired item can be selected by touch panel operation, and the setting of each item can be changed. Similarly, items 805 to 809 are setting items for the second imaging unit, and different settings are possible for the first and second imaging units. In manual mode, all setting items can be set, but the items that can be set are set automatically depending on the mode. For example, in aperture priority mode, the aperture value can be set, but the shutter speed is set automatically, so manual setting is not accepted.

[0074] Although this embodiment was described using Figure 14, the imaging device may be equipped with all of these modes, or it may be equipped with some of these modes. Furthermore, the modes described in Figures 4 and 7 in Embodiments 1 and 2 may be combined.

[0075] In the above-described embodiments 1 to 3, an example of the screen display during monitoring when shooting still images or videos will be explained with reference to Figure 16. Figure 16(A) is a single-screen display, showing an image from either the first or second imaging unit. In Figure 16(A), 8 is the display, and 92 is the mode display button, which displays the currently set mode name on the OSD. 93 is the camera setting button, and selecting this displays the menu screen shown in Figure 8. This selection operation is performed, for example, by a touch panel built into the display 8. 94 is the screen switching button, which switches between the first and second imaging units in single-screen display, and between single-screen and double-screen display. Switching between the first and second imaging units in single-screen display, and switching between single-screen and double-screen display, may be done sequentially by pressing the same button, or switching between the first and second imaging units in single-screen display, and switching between single-screen and double-screen display, may be done by different operation buttons.

[0076] Figure 16(B) shows a dual-screen display, showing images from both the first and second imaging units simultaneously. Depending on the imaging mode, it may be possible to switch between displaying only one screen or to allow selection between one and two screens. For example, in Normal Shooting 1 in Figure 4, only the first imaging unit is active, so only a single-screen display is necessary. In this case, the screen switching button is not displayed. On the other hand, each mode in Figure 14 uses both the first and second imaging units, so the screen switching button is displayed. In this way, the usability is improved by selectively switching between single and two screen displays depending on the shooting mode.

[0077] As described above, this embodiment, like embodiments 1 and 2, enables multiple imaging units to take images in conjunction with a single shooting command from the operating means, allowing for simultaneous control of multiple imaging units with simple operation and enabling shooting in multiple shooting modes, thus providing a user-friendly imaging device. [Examples]

[0078] This embodiment describes an embodiment using a first imaging unit and a second imaging unit with different hardware configurations. That is, while in Embodiments 1 to 3 the first imaging unit and the second imaging unit have basically the same hardware configuration, this embodiment uses different hardware configurations for the first imaging unit and the second imaging unit.

[0079] Figure 17 is an explanatory diagram of the imaging modes in this embodiment. Similar to Figures 4, 7, and 14, Figure 17 shows the operation of the two imaging units, the recorded image, and an overview of the control in each mode. There are five imaging modes shown in Figure 17: imaging / recognition, visible / near-infrared, telephoto / wide-angle, panoramic, and with / without OLPF, each of which will be explained below.

[0080] First, in the imaging / recognition shooting mode, normal imaging is performed in the first imaging unit, and image recognition processing is performed in the second imaging unit. Detection such as AE, AF, and AWB may be performed in the second imaging unit. The second imaging unit may also be used as a dedicated sensor for phase-detection AF. In this shooting mode, the same effects as the normal imaging mode of Example 1 can be obtained.

[0081] Next, in the visible / near-infrared imaging mode, the first imaging unit captures visible light, and the second imaging unit captures near-infrared light, or near-infrared light to infrared light. The image signal processing circuit 5 combines the imaging signals from the first and second imaging units. Since the first imaging unit captures visible light, an image with good color reproduction is obtained, and since the second imaging unit captures near-infrared light, or near-infrared light to infrared light, a highly sensitive image is obtained. This imaging mode has the effect of achieving both high color reproduction and high sensitivity.

[0082] Next, in tele / wide-angle shooting mode, images are captured using lenses with different focal lengths in the first and second image units. Alternatively, images with equivalently different focal lengths are captured in the first and second image units by combining electronic zoom and lenses. During recording, each image captured by the first and second image units is recorded. Images captured at the same time by the first and second image units may be recorded together. In this shooting mode, since two images with equivalently different focal lengths can be captured simultaneously by using lenses with different focal lengths or by combining electronic zoom and lenses, it has the effect of simultaneously capturing wide-angle and narrow-angle images, or images of the entire scene and a specific part of the scene.

[0083] Next, in panoramic shooting mode, a first and second imaging unit are used to capture a wider angle than when shooting with a single imaging unit, and this is referred to here as panorama. The first and second imaging units are positioned so that their imaging ranges touch or partially overlap. For example, the first imaging unit captures a 90-degree range to the left, and the second imaging unit captures a 90-degree range to the right, and the imaging units are set up horizontally so that their respective imaging ranges touch. By combining the images captured by the first and second imaging units using an image signal processing circuit, it is possible to capture a 180-degree wide-angle image. This shooting mode is effective because it allows for the generation of a wide-angle panoramic image in a single capture.

[0084] Finally, in the OLPF-enabled / disabled imaging mode, the first imaging unit uses an imaging unit equipped with an optical low-pass filter, and the second imaging unit uses an imaging unit without an optical low-pass filter, capturing images simultaneously with both units. The images captured by the first and second imaging units are recorded separately. When imaging with an optical low-pass filter, aliasing noise that occurs when sampling the optical image at the pixels during imaging can be removed. This reduces so-called moiré patterns that occur when photographing subjects with fine details. However, using an optical low-pass filter results in a decrease in resolution due to the characteristics of the optical low-pass filter. On the other hand, when imaging with an imaging unit without an optical low-pass filter, moiré patterns may occur depending on the subject, but there is an advantage in obtaining high-resolution images. In this imaging mode, images with less moiré and high-resolution images can be captured simultaneously, allowing the photographer to select the image that best suits their intentions depending on the subject.

[0085] Furthermore, the contents of Examples 1 to 3 may be combined in each shooting mode of this embodiment.

[0086] Figure 18 is an external view showing an example of the imaging device in this embodiment, where (A) is a rear view and (B) is a front view. The difference from Figure 3 is that in Figure 3, the lens section 2a and lens section 2b are arranged horizontally, whereas in Figure 18 they are arranged vertically. Lens section 2a may be a fixed lens and lens section 2b may be a detachable lens.

[0087] Figure 19 is an external view showing another example of the imaging device in this embodiment, and shows a rear view. The difference from Figure 2 is that in Figure 2, the lens unit 2a and lens unit 2b are arranged horizontally, whereas in Figure 19 they are arranged vertically. Similarly, different imaging units with different focal lengths may be used for lens unit 2a and lens unit 2b. As in Figure 18, lens unit 2a may be a fixed lens and lens unit 2b may be a detachable lens.

[0088] When lens units 2a and 2b are arranged vertically, there is the advantage that no horizontal parallax occurs between the two imaging units. However, this is not suitable for applications where parallax between the two imaging units is desired, such as 3D imaging or panoramic imaging.

[0089] As described above, this embodiment, like Embodiment 1, enables multiple imaging units to take images in conjunction with a single shooting instruction from the operating means, and allows for simultaneous control of multiple imaging units with simple operation to take images in multiple shooting modes, thus providing an easy-to-use imaging device. [Examples]

[0090] This embodiment illustrates another appearance of the imaging device.

[0091] Figure 20 is an external view showing an example of the imaging device in this embodiment, and shows a rear view. The difference from Figure 19 is that the number of lens units has been increased from two to three. Corresponding to the lens units, three imaging units are also required: imaging units 3a, 3b, and 3c (drawings omitted). For example, the operations described in Embodiments 1 to 4 are performed by lens units 2b and 2c, and normal shooting is performed by lens unit 2a. Still image and video capture by lens unit 2a and distance detection and recognition processing by lens units 2b and 2c can be performed independently, enabling faster and higher-performance imaging.

[0092] As described above, in this embodiment, an imaging device equipped with multiple imaging units is provided with means for individually setting the control settings for each imaging unit, means for interlocking and controlling the multiple imaging units, and operating means. This makes it possible to take images in conjunction with the multiple imaging units in response to a single shooting command from the operating means. [Examples]

[0093] Figure 21 is a block diagram showing the configuration of the imaging device in this embodiment. In Figure 21, functions similar to those in Figure 1 are denoted by the same reference numerals, and their explanations are omitted. The difference between Figure 21 and Figure 1 is that Figure 21 includes an image and sound signal processing circuit 20, a microphone 16, a speaker 17, and a proximity sensor 18, and that lens units 2a and 2b are lenses with different focal lengths. The focal lengths of lens units 2a and 2b are arbitrary, but lens unit 2b is narrow-angle relative to lens unit 2a, and lens unit 2a is wide-angle relative to lens unit 2b. In addition, the size, pixel count, and other specifications of imaging units 3a and 3b may be the same or different, but imaging unit 3a is a wide-angle image imaging unit, and 3b is a narrow-angle image imaging unit.

[0094] The image and audio signal processing circuit 20, in addition to the image processing performed by the image signal processing circuit 5 in Figure 1, also has an audio signal processing function. It performs predetermined audio signal processing, such as encoding, on the audio input from the microphone 16. It also decodes the encoded audio signal and outputs it from the speaker 17.

[0095] In addition to the functions described in Figure 1, the control circuit 6 is connected to a proximity sensor 18 and controls the connected proximity sensor 18 according to the operating mode. The control circuit 6 also controls the display 8, encoding / decoding circuit 15, and recording / playback circuit 10 either via the image / audio signal processing circuit 20 or directly.

[0096] In this embodiment, the control circuit 6 controls the encoding and decoding circuit 15, the imaging units 3a and 3b, the image and audio signal processing circuit 20, the display 8, the recording and playback circuit 10, etc., according to the operation mode set in advance by user operation. With this control, it is possible to control the imaging units 3a and 3b independently to take images, to control the imaging units 3a and 3b in conjunction to take images, to control the imaging units 3a and 3b simultaneously to take still images and videos at the same time, to take still images with the other imaging unit while one imaging unit is taking video and recording, or to take video simultaneously. Furthermore, it is possible to control the system to display or record either the image signal captured by the imaging unit 3a, the image signal captured by the imaging unit 3b, or the image signal generated by combining the image signals captured by the imaging units 3a and 3b.

[0097] An example of the imaging device in this embodiment is shown in Figure 18, for example, in which the lens units 2a and 2b are arranged vertically, and a display 8 with an integrated touch panel is located on the back side. The lens units 2a and 2b are arranged at a predetermined interval so that they each have approximately the same optical axis. The lens units 2a and 2b may be arranged side by side, fixed, or detachable. In addition to the touch panel described above, user operation may also be performed by hardware means such as a shutter button.

[0098] Next, details of the shooting modes in the present embodiment will be described. FIG. 22 and FIG. 23 are lists of shooting modes in the present embodiment, where FIG. 22 is a list of normal shooting modes, and FIG. 23 is a list of simultaneous shooting modes. The normal shooting modes in FIG. 22 are modes in which shooting is performed with either one of the imaging units 3a and 3b, including the still image shooting mode of Mode 1 and the moving image shooting mode of Mode 2. The simultaneous shooting modes in FIG. 23 are operation modes in which a wide-angle image and a narrow-angle image are simultaneously shot by the imaging units 3a and 3b, including simultaneous moving image and still image shooting of Mode 3, simultaneous still image shooting of Mode 4, and simultaneous moving image shooting of Mode 5. The tables in FIG. 22 and FIG. 23 have, in the leftmost vertical column from the top, a shooting mode, a wide-angle imaging unit, a narrow-angle imaging unit, and an operation described, and for shooting modes 1 to 5, the distinction between still images and moving images shot by the narrow-angle imaging unit and the wide-angle imaging unit, and an overview of the operation of each mode are described.

[0099] First, the still image shooting mode of Mode 1 in FIG. 22 will be described. This mode is a mode for shooting a wide-angle or narrow-angle still image using either one of the two imaging units 3a or 3b. Which imaging unit is used depends on the zoom magnification set by a zoom operation. Zooming includes optical zoom that performs zooming with a zoom lens, and electronic zoom that performs enlargement processing through signal processing, and any method including a combination of both methods may be adopted, but in the following description, the lens units 2a and 2b are fixed-focus lenses, and zoom processing is performed by electronic zoom. Let the focal lengths of the wide-angle lens unit 2a and the narrow-angle lens unit 2b be fa and fb respectively. Since the wide-angle side has a smaller focal length, fa < fb. If the state of the focal length fa is defined as a zoom magnification of 1×, and the electronic zoom magnification by the imaging unit 3a is defined as Ma, the electronic zoom magnification may be set within the range of 1 ≦ Ma < (fb / fa). Further, if the electronic zoom magnification by the narrow-angle imaging unit 3b is defined as Mb, the electronic zoom magnification may be set within the range of 1 ≦ Mb < Mm. Here, Mm is the maximum value of the electronic zoom magnification for the imaging unit 3b. The value of Mm is arbitrary, but since the resolution decreases when the electronic zoom magnification is increased, the maximum value Mm may be determined within the allowable range of image quality.

[0100] FIG. 24 shows the relationship between the focal length and the zoom magnification when the zoom magnification is defined with reference to the wide-angle imaging unit 3a. As shown in FIG. 24, when zooming the focal length f in the range of fa≦f<fb, imaging is performed by the wide-angle imaging unit 3a. During this period, the electronic zoom magnification Ma of the wide-angle imaging unit 3a is controlled within the above range. When zooming the focal length f in the range of fb≦f<fm, imaging is performed by the narrow-angle imaging unit 3b. During this period, the electronic zoom magnification Mb of the narrow-angle imaging unit 3b is controlled within the aforementioned range. If the zoom magnification at this time is defined with the focal length fa of the wide-angle imaging unit as the reference (1x) for the zoom magnification, continuous zooming from 1x to fb / fa times by the wide-angle imaging unit and from fb / fa times to fm / fa times by the narrow-angle imaging unit can be achieved. For example, assuming that the focal length fa expressed in 35mm equivalent is 20mm and fb is 50mm, the electronic zoom of the wide-angle imaging unit may be controlled within the range of 1x to 2.5x. If fm is set to be equivalent to, for example, 200mm, the electronic zoom for the narrow-angle imaging unit may be controlled within the range of 1x to 4x, and the maximum combined zoom magnification of the wide-angle imaging unit and the narrow-angle imaging unit is 10x. In the above description, the zoom is assumed to be performed by electronic zoom, but an optical zoom using a zoom lens may also be adopted.

[0101] Figure 25 is an explanatory diagram of the single-screen display in Mode 1 still image shooting mode. Figure 25(A) shows an example of displaying a wide-angle image captured by the imaging unit 3a, and Figure 25(B) shows an example of displaying a narrow-angle image captured by the imaging unit 3b. In Figures 25(A) and (B), 8 is the display screen. 52 is the subject, a person. 53 to 56 are shooting mode setting icons. 53 is the still image shooting mode icon, 54 is the video shooting mode icon, 55 is the simultaneous video shooting mode icon, and 56 is the simultaneous still image shooting mode icon. The user can set the shooting mode by touching the desired shooting mode icon. This setting is performed by the control circuit 6 (Figure 21) detecting user operation via the touch panel and controlling the imaging units 3a and 3b, the interface circuit 4, and the image and audio signal processing circuit 20. In the example in Figure 25, the still image shooting mode icon 53 is selected. The icon for the selected mode is displayed separately from the icons for unselected modes, such as by increasing its brightness, so that it is possible to distinguish which icon is selected. 500 is the still image capture button; touch this button to capture and record an image. In the preparation state before shooting, as already explained, the moving image being captured by the imaging unit 3a or 3b is displayed in real time on the image display unit 50 on the display. In Figure 25(A), 51 is a frame representing the narrow-angle image area, allowing you to understand the shooting angle of the narrow-angle image while monitoring the wide-angle image. Note that the frame representing the narrow-angle image area 51 may be switched off to not be displayed. Alternatively, the narrow-angle image may be displayed in a sub-screen. 601 is the playback mode button; touch this button to switch to the playback mode, which will be described later.

[0102] Switching between the wide-angle and narrow-angle image displays in Figures 25(A) and (B) is done by touching the field of view switching buttons 501 or 502 on the display. Zoom operations are performed by pinching in and out on the touch panel. For continuous zooming, use pinch in and pinch out; for instantaneous switching between wide-angle and narrow-angle images, use the field of view switching buttons 501 and 502.

[0103] In the narrow-angle image display in Figure 25(B), 58 represents the wide-angle shooting area, which is superimposed on the narrow-angle image in a reduced size. When taking a still image, the narrow-angle image is captured and recorded by touching the shutter button, similar to the wide-angle image display in Figure 25(A). The shutter button operation may be controlled in two stages by detecting the pressure applied to the touch panel. When the pressure is weak, the shutter button is determined to be half-pressed, and when the pressure exceeds a predetermined value, it is determined to be fully pressed. Exposure control and focus control may be performed in the half-pressed state, and the shooting operation in the fully pressed state.

[0104] In this Mode 1 still image shooting mode, the camera automatically switches between image units 3a and 3b when zooming using the angle of view switching button or pinch-in / out gestures. This allows the user to operate the device with the same ease as a camera with a single image unit, making it user-friendly.

[0105] Next, we will explain the video recording mode in Mode 2. Figure 26 is an explanatory diagram of the single-screen display in video recording mode, with Figure 26(A) showing an example of displaying a wide-angle image captured by the imaging unit 3a and Figure 26(B) showing an example of displaying a narrow-angle image captured by the imaging unit 3b. Except for the difference that Figure 25 is the still image shooting mode and Figure 26 is the video recording mode, switching between wide-angle and narrow-angle images, and zooming operations using pinch-in and pinch-out are almost the same as in the still image shooting mode example in Figure 25. In this video recording mode, the video recording mode icon 54 is selected in the shooting mode setting icon. To shoot video, touch the video recording button 600. When the video recording button 600 is touched, the control circuit 6 detects this, as in the case of still image shooting, and controls the imaging unit, signal processing unit, etc., to perform operations such as capturing video, signal processing, and recording. The video recording button 600 is distinguished from the still image recording button 500 in Figure 25 by changing the border of the button.

[0106] Figure 27 shows the display screen in video recording mode, specifically the display screen while recording a wide-angle image. Figure 27(A) shows the screen for the wide-angle image, and Figure 27(B) shows an example of the screen display when switching to a narrow-angle image display using the field of view switching button 501 while shooting a wide-angle image. In Figure 27(A), since it is the screen during video recording, the video recording end button 700 is displayed. Also, 70 is the elapsed time display, showing the time elapsed since the start of recording. To end video recording, touch the video recording end button 700. Note that in the narrow-angle image display in Figure 27(B), recording of the narrow-angle image is not performed, so the video recording end button is not displayed. However, still image shooting is possible, and by touching the still image shooting button 500, it is possible to take a still image of the narrow-angle image while shooting a video of the wide-angle image. Furthermore, at this time, a reduced image 701 of the wide-angle image, which is a scaled-down version of the wide-angle image display similar to that in Figure 27(A), is superimposed to indicate that wide-angle video is being recorded. Also, 503 is a display mode icon that indicates which mode the displayed screen is in. In the case of Figure 27(A), it indicates that it is in video recording mode, and in the case of Figure 27(B), it indicates that it is in still image shooting mode, which allows for still image capture of narrow-angle images.

[0107] In the wide-angle video recording screen shown in Figure 27(A), when zooming in is performed by pinching out, electronic zoom processing is executed, and recording continues with the zoom level adjusted accordingly. When zooming in further reaches a zoom magnification equivalent to the focal length of the narrow-angle image (point B in Figure 24), two actions are possible. One is to continue capturing images with the wide-angle image capture unit 3a, and the other is to hand over the video recording operation to the narrow-angle image capture unit 3b and continue recording. Either method may be adopted.

[0108] In Figure 27(A), it is possible to capture and record wide-angle still images while recording wide-angle video. However, because wide-angle video is being captured and recorded, there are limitations on the image size of the still images to be recorded. In contrast, since the narrow-angle imaging unit is not performing any shooting or recording operations, it is possible to capture and record high-quality narrow-angle still images while recording wide-angle video.

[0109] Figure 28 shows the display screen in video recording mode. While Figure 27 shows the display screen during wide-angle image recording, Figure 28(A) shows the display screen during narrow-angle image recording. To end recording of the narrow-angle image, simply touch the video recording end button 700. The angle of view switching button 502 switches to the screen shown in Figure 28(B), allowing you to display the wide-angle image. Since the wide-angle image in Figure 28(B) is not being recorded, the video recording end button is not displayed. However, at this time, it is possible to take a still image of the wide-angle image while a video of the narrow-angle image is being recorded.

[0110] In the Mode 2 video recording mode described above, it is possible to capture narrow-angle still images at any time while shooting and recording wide-angle video, and similarly, it is possible to capture wide-angle still images at any time while shooting and recording narrow-angle video. In this way, it is possible to capture still images while shooting and recording video, and this video recording mode also serves as the simultaneous video and still image shooting mode of Mode 3 shown in Figure 23.

[0111] Furthermore, it is desirable to group still images taken during video recording so that they can be played back together in association with the video. For this reason, information indicating that the recorded video and still image files belong to the same group may be recorded, or the timing of when the still image was taken may be recorded in the video.

[0112] Next, we will explain the simultaneous still image shooting mode of Mode 4 shown in Figure 23. Figure 29 is an explanatory diagram of the display screen in the simultaneous still image shooting mode. In Figures 29(A) and (B), the simultaneous still image shooting mode icon 56 is selected, and the screen display content is the same as in the still image shooting mode in Figure 25. In the still image shooting mode, when you touch the shutter button to take a picture, you take either a wide-angle or narrow-angle still image. In contrast, in this simultaneous still image shooting mode, you can take and record both wide-angle and narrow-angle still images simultaneously with a single shutter button operation. It is desirable to group the wide-angle and narrow-angle still images taken simultaneously so that they can be played back together during playback. For this reason, information indicating that they belong to the same group may be recorded in the recorded video and still image files.

[0113] Furthermore, if the display screens for Mode 1 (still image shooting mode) and Mode 4 (simultaneous still image shooting mode) are similar, it may be difficult for the user to tell which mode is selected. Therefore, in addition to the shooting mode icon, it may be possible to make it easier to recognize the current setting mode by, for example, differentiating the shape of the shutter button or the shutter sound for the two modes. Also, in order to speed up recording and playback, multiple recording and playback circuits and recording media may be provided, and wide-angle still images and narrow-angle still images may be recorded on different recording media.

[0114] Figure 30 is an explanatory diagram of the two-screen display in the still image simultaneous shooting mode. This display method simultaneously displays the wide-angle image and narrow-angle image screens from Figures 29(A) and 29(B) side by side. This display can be switched to the two-screen display of Figure 30(A) by, for example, touching the narrow-angle image area 51 on the screen of Figure 29(A) and moving to the edge of the screen, or by providing a dedicated display switching button (not shown in the figure).

[0115] Figure 30(A) shows the image in a vertical orientation, while Figure 30(B) shows the image scaled down to its original aspect ratio. In this case, related shooting information may be displayed in the display space. In Figure 30(b), the focal length display 210, exposure level display 211, and level indicator display 212 are shown as related shooting information, but other arbitrary shooting information may also be displayed.

[0116] Figure 31 is an explanatory diagram of the single-screen display in the simultaneous video recording mode of Mode 5 described in Figure 23. Figure 31(A) is the display screen for the wide-angle image, and Figure 31(B) is the display screen for the narrow-angle image. The screen display content is almost the same as the display for the simultaneous still image recording mode in Figure 29, except that the mode icon is the simultaneous video recording mode icon 55 and the video recording button 600 is touched to start video recording. In this shooting mode, when video recording is started by touching the video recording button 600, both the wide-angle imaging unit 3a and the narrow-angle imaging unit 3b simultaneously capture and record wide-angle and narrow-angle videos. In addition, to speed up recording and playback, multiple recording and playback circuits and recording media may be provided, and wide-angle and narrow-angle videos may be recorded on different recording media.

[0117] Figure 32 shows the display screen during simultaneous video recording in video recording mode; Figure 32(A) is the display screen for the wide-angle image, and Figure 32(B) is the display screen for the narrow-angle image. Still image capture is also possible during video recording. However, since both the wide-angle and narrow-angle imaging units are in video recording operation, the size and timing of still images to be captured are limited.

[0118] Figure 33 is an explanatory diagram of the single-screen display in Mode 3, the simultaneous video and still image shooting mode described in Figure 23. The simultaneous video and still image shooting mode has already been explained as an operation within the video shooting mode using Figures 27 and 28, but Figure 33 is an example in which a dedicated icon 57 for the simultaneous video and still image shooting mode is provided. When a dedicated mode for simultaneous video and still image shooting is provided in this way, the still image shooting function within the video shooting mode may be omitted.

[0119] In the embodiments described above, an imaging device having the five shooting modes shown in Figures 22 and 23 was described. However, it is not necessary to have all of these shooting modes. For example, it may only have mode 1, the still image shooting mode, and mode 4, the simultaneous still image shooting mode, which are related to still images. Alternatively, it may have modes 1, 2, 3, and 4, or any one or more of these five modes.

[0120] Next, the playback mode of this embodiment will be described. Figure 34 is an explanatory diagram of the playback mode screen display in this embodiment. To display the playback mode screen, for example, touch the playback mode button 601 from the shooting mode screen such as Figure 25, and the control circuit will detect this and switch to playback mode. In Figure 34(A), 150, 151, 160-164 are thumbnail images, which are thumbnail images that are reduced data of multiple image data recorded on the recording medium, arranged in two dimensions and displayed. The arrangement order of each image is in order of recording time. The images of the thumbnail images in the figure are not shown, but in reality, each thumbnail image is played back and displayed as a reduced image. 152 is a still image icon, and 153 is a video icon, indicating whether the image data corresponding to each thumbnail is a still image or a video. In Figure 34, icons to distinguish between still images and videos are added to the thumbnails, but other display methods that can distinguish between still images and videos may be adopted.

[0121] On the screen in Figure 34(A), touching the display mode button 154 switches to the screen in Figure 34(B). In Figure 34(B), 156 is the still image simultaneous shooting mode icon, 157 is the video simultaneous shooting mode icon, and 159 is the video and still image simultaneous shooting mode icon, indicating the shooting mode in which the image corresponding to each thumbnail was taken. Thumbnail 166 in Figure 34(B) corresponds to the two thumbnails 160 and 161 in Figure 34(A). These images correspond to two images taken in the still image simultaneous shooting mode. Similarly, thumbnails 167 and 169 in Figure 34(B) correspond to thumbnails 162 and 151, and 164 and 165, respectively, in Figure 34(A).

[0122] Figure 35 is an explanatory diagram of another display method for the playback mode screen in this embodiment. Figure 35(A) is another display method that shows the same content as Figure 34(B), but with the area of ​​the narrow-angle image and icons to distinguish between video and still images added to the thumbnail of the wide-angle image. Thumbnail images 166, 167, and 169 in Figure 34(B) correspond to thumbnail images 176, 177, and 179 in Figure 35(A), respectively.

[0123] Figure 35(B) is a representative screen of the wide-angle video corresponding to thumbnail image 177, which is displayed when thumbnail image 177 is selected in Figure 35(A). Touching the play button 180 will play this wide-angle video. 181 is a frame representing the narrow-angle image area, and a narrow-angle video play button 182 may be displayed, allowing direct playback of the narrow-angle video by touching it. Similarly, for still images, they are first displayed in wide-angle mode, and the narrow-angle still image is played using the narrow-angle image play button.

[0124] In Figures 34(B) and 35(A), icons indicating the shooting mode are added to each thumbnail for display purposes. However, instead of this display, the shooting mode information for a specific thumbnail image may be displayed when that thumbnail is selected.

[0125] As described above, this embodiment provides an imaging device equipped with a wide-angle image capturing unit and a narrow-angle image capturing unit, in which the imaging operation is controlled to be performed using a first shooting mode for capturing narrow-angle or wide-angle still images, a second shooting mode for capturing narrow-angle or wide-angle video, a third shooting mode for simultaneously capturing narrow-angle and wide-angle still images, and a fourth shooting mode for simultaneously capturing wide-angle video and narrow-angle still images, or narrow-angle video and wide-angle still images. Furthermore, the user can select from these shooting modes through user settings.

[0126] That is, an imaging device having a first imaging unit for capturing narrow-angle images, a second imaging unit for capturing wide-angle images, a signal processing unit, a control unit, an operation input unit, and a recording unit, wherein the signal processing unit generates first still image data and first video data from the signal captured by the first imaging unit, and generates second still image data and second video data from the signal captured by the second imaging unit, and the control unit controls the first imaging unit, the second imaging unit, the signal processing unit, and the recording unit according to the shooting mode input from the operation input unit. The system controls the recording function and is configured to have a shooting mode that includes a first shooting mode for recording either a first still image or a second still image, a second shooting mode for recording either a first video or a second video, a third shooting mode for recording either a second still image while recording the first video, or recording a first still image while recording the second video, and a fourth shooting mode for simultaneously recording both the first and second still image.

[0127] This allows users to select their desired shooting mode through user settings, enabling them to control multiple imaging units independently or simultaneously for shooting with simple operation, thus providing a user-friendly imaging device. [Examples]

[0128] Figure 36 is an overview view of the imaging device in this embodiment. In Figure 36, (A) is a rear view, (B) is a right side view, and (C) is a front view. A display 8 with an integrated touch panel is located on the rear side of the imaging device. The difference between the imaging device in this embodiment and that in Embodiment 6 is that a 2c is added to the lens units 2a and 2b, and a 3c is added to the imaging units 3a and 3b. In Figure 36, the lens units 2a and 2c are ultra-wide-angle lenses, and a so-called 360-degree spherical image can be generated by combining the images captured by the imaging units 3a and 3c. In this embodiment, the 360-degree spherical image captured by the imaging units 3a and 3c is used as the wide-angle image, and the image captured by the imaging unit 3b is used as the narrow-angle image. The other parts are the same as in Embodiment 6 and will not be described. [Examples]

[0129] Figure 37 is an overview of the imaging device in this embodiment. In Figure 37, (A) is a rear view, (B) is a right side view, and (C) is a front view. Similar to Figure 36, a display 8 with an integrated touch panel is located on the rear side of the imaging device.

[0130] This embodiment, like Embodiment 7, uses a 360-degree spherical image capture unit as the wide-angle image capture unit. The difference from Embodiment 7 is that the lens unit 2b is provided on the side of the main body, as shown in Figure 37(B). With the configuration of this embodiment, the lens of the narrow-angle image capture unit is at approximately a 90-degree angle to the lens of the wide-angle image capture unit, which has the advantage of making it easier for the narrow-angle image capture unit to capture the blind spot of the wide-angle image capture unit, and also makes it less likely for the photographer to obstruct the shooting of wide-angle images when shooting with the wide-angle image capture unit.

[0131] Furthermore, the display 8 may be movable. Figure 38 is a side view of the main body from the left side, showing the display in Figure 37 made movable, with the display 8 mounted on the movable part 185 and rotated approximately 90 degrees to the side. Making the display movable in this way makes shooting operations easier. Additionally, an electronic viewfinder 186 may be provided on the side of the main body. The lens part 2b is provided on the side opposite to the electronic viewfinder 186. The other parts are the same as in Embodiment 6 and will not be described.

[0132] Figure 39 is an explanatory diagram of the shooting screen in this embodiment. In the shooting mode of this embodiment, the same display as the shooting mode described in Embodiment 6 may be used, but the two-screen display shown in Figure 39 may also be used. In Figure 39(A), 190 is the display area for the narrow-angle image, and 192 is the display area for the 360-degree image, which is also the display area for the wide-angle image. The wide-angle image display area displays the 360-degree image captured by the imaging units 3a and 3c and processed for synthesis. The narrow-angle image display area 190 displays the narrow-angle image captured by the imaging unit 3b. The display of the 360-degree image in the 360-degree image display area 192 allows the 360-degree image to be converted into an image viewed from any viewpoint by dragging. When an image corresponding to the narrow-angle image display area 190 is displayed in the 360-degree image display area 192, the area of ​​the corresponding narrow-angle image is displayed by frame 191.

[0133] Figure 39(B) shows an example of the display when the 360-degree image display area 192 in Figure 39(A) is enlarged by a pinch-out gesture. For example, area 193 in Figure 39(A) is enlarged and displayed in area 195 in Figure 39(B).

[0134] Although embodiments have been described above, the present invention is not limited to the embodiments described above and includes various modifications. For example, the embodiments described above are described in detail to make the present invention easier to understand and are not necessarily limited to those having all the configurations described. Furthermore, it is possible to replace parts of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add configurations from other embodiments to the configuration of one embodiment. In addition, it is possible to add, delete, or replace parts of the configuration of each embodiment with other configurations. [Explanation of Symbols]

[0135] 1: Imaging device, 2a, 2b, 2c: Lens unit, 3a, 3b, 3c: Imaging unit, 4: Interface circuit, 5: Image signal processing circuit, 6: Control circuit, 7: Attitude detection unit, 8: Display, 9: Operation input unit, 10: Recording and playback circuit, 11: Recording medium, 12: Memory, 13: Position information input unit, 14: Wireless communication unit, 15: Encoding and decoding circuit, 16: Microphone, 17: Speaker, 18: Proximity sensor, 20: Image and audio signal processing circuit, 31: Mode setting dial, 32: Shutter button, 33: Operation button, 34: Video recording button, 35: Power switch, 50: Image display unit, 51: Narrow-angle image area, 52: Subject, 53: Still image shooting mode icon, 54: Video recording mode icon, 55: 56: Simultaneous still image shooting mode icon, 92: Mode display button, 93: Camera settings button, 94: Screen switching button, 140: Video display area, 141: Input image of person, 142: Input image of background, 143: Distance histogram display area, 144: Selection frame, 145: Image display area, 146: Image quality adjustment display, 800~804: Settings for the first image sensor, 805~809: Settings for the second image sensor, 186: Electronic viewfinder, 190: Narrow-angle image display area, 192: Spherical image display area, 500: Still image shooting button, 501, 502: Angle of view switching button, 600: Video recording button, 601: Playback mode button, 700: Video recording end button

Claims

[Claim 1] An imaging device, The imaging device includes a first imaging unit positioned on the first side for capturing narrow-angle images, The imaging device includes a second imaging unit positioned on the first side and capturing an image through a first ultra-wide-angle lens, A third imaging unit is positioned on the second side opposite to the first side of the imaging device, and captures an image through a second ultra-wide-angle lens. A signal processing unit configured to generate first still image data or first video data from a signal acquired by the first imaging unit, second still image data or second video data from a signal acquired by the second imaging unit, and third still image data or third video data from a signal acquired by the third imaging unit, Display unit and Records Department, The system comprises a control unit configured to control the first imaging unit, the second imaging unit, the third imaging unit, the signal processing unit, the display unit, and the recording unit, The control unit, An imaging device characterized in that, when recording the second video data, based on an image acquired by the second imaging unit through the first ultra-wide-angle lens, to the recording unit by combining it with the third video data, based on an image acquired by the third imaging unit through the second ultra-wide-angle lens, upon receiving an input for a field of view switching operation, the device controls itself to continue recording the combined video data, consisting of the second video data and the third video data acquired by the second imaging unit, to the recording unit, while switching the field of view in response to the input for the field of view switching operation.

Images