Imaging device, information processing method, and program

Abstract

The present disclosure relates to an imaging device, an information processing method, and a program that allow the disparity between a plurality of images having different individual eye optical systems as the viewpoints thereof to be more easily ascertained. One image among a plurality of images that have a plurality of individual eye optical systems as the viewpoints thereof is selectively displayed while being dynamically switched, said individual eye optical systems having optical paths that are independent from one another. The present disclosure can be applied, for example, to an imaging device, an electronic apparatus, an interchangeable lens or a camera system in which a plurality of individual eye lenses extend, an information processing method, a program, or similar.

Description

[Technical Field]

[0001] The present disclosure relates to an imaging device, an information processing method, and a program, and more particularly to an imaging device, an information processing method, and a program that enable easier understanding of the parallax between multiple images taken from different unit optical systems. [Background technology]

[0002] Conventionally, in imaging devices that capture images using multiple unitary optical systems with independent optical paths, methods have been considered for displaying the entire area of ​​the captured image or for displaying a unitary image with the unitary optical system as the viewpoint (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2019 / 065260 Summary of the Invention [Problem to be solved by the invention]

[0004] However, in the case of the method described in Patent Document 1, it is difficult to intuitively grasp the degree of parallax with which the subject is imaged.

[0005] The present disclosure has been made in consideration of such circumstances, and makes it possible to more easily grasp the parallax between a plurality of images taken from different unit optical systems as viewpoints. [Means for solving the problem]

[0006] According to one aspect of the present technology, there is provided an imaging device in which optical paths are independent of each other. 5 or more Viewpoint of unitary optical system 5 or more Image Select some or all of the five or more images one by one in the order that the scanning trajectory becomes linearly symmetric or rotationally symmetric in the relative positional relationship of each viewpoint of the five or more images. The imaging device is equipped with a display control unit that dynamically switches between images displayed on the display unit.

[0007] In one aspect of the present technology, the information processing method includes: 5 or more Viewpoint of unitary optical system 5 or more Image Select some or all of the five or more images one by one in the order that the scanning trajectory becomes linearly symmetric or rotationally symmetric in the relative positional relationship of each viewpoint of the five or more images. This is an information processing method in which information is dynamically switched and displayed on a display unit.

[0008] The program according to one aspect of the present technology is a program for causing a computer to 5 or more Viewpoint of unitary optical system 5 or more Image Select some or all of the five or more images one by one in the order that the scanning trajectory becomes linearly symmetric or rotationally symmetric in the relative positional relationship of each viewpoint of the five or more images. This is a program that functions as a display control unit that dynamically switches between images and displays them on the display unit.

[0009] In the imaging device, the information processing method, and the program according to one aspect of the present technology, the optical paths are independent of each other. 5 or more Viewpoint of unitary optical system 5 or more Image Some or all of the five or more images are selected one by one in the order in which the scanning trajectory is linearly symmetric or rotationally symmetric in the relative positional relationship of each viewpoint of the five or more images. The images are dynamically switched and displayed on the display unit. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is a perspective view showing an example of the configuration of an embodiment of a camera to which the present technology is applied; [Figure 2] 1 is a diagram illustrating an example of the configuration of an embodiment of a camera to which the present technology is applied. [Figure 3] FIG. 2 is a block diagram showing an example of the electrical configuration of the camera. [Figure 4] FIG. 1 is a diagram illustrating an example of a three-chip image sensor. [Figure 5] FIG. 10 is a diagram illustrating an example of an entire image. [Figure 6] FIG. 10 is a diagram illustrating an example of an ommatidium image. [Figure 7] FIG. 10 is a diagram illustrating an example of a composite image. [Figure 8] FIG. 2 is a diagram illustrating an example of the positional relationship between a camera and a subject. [Figure 9] FIG. 10 is a diagram showing an example of an image cut out from an ommatidium in the entire image. [Figure 10] 10A and 10B are diagrams illustrating an example of the display order of ommatidial cutout images. [Figure 11] FIG. 10 is a diagram illustrating an example of a displayed image. [Figure 12] FIG. 10 is a diagram illustrating an example of a shift. [Figure 13] FIG. 10 is a diagram showing an example of an image cut out from an ommatidium in the entire image. [Figure 14] FIG. 10 is a diagram illustrating an example of a displayed image. [Figure 15] 10A and 10B are diagrams illustrating an example of the display order of ommatidial cutout images. [Figure 16] 10A and 10B are diagrams illustrating an example of the display order of ommatidial cutout images. [Figure 17] 10A and 10B are diagrams illustrating an example of the display order of ommatidial cutout images. [Figure 18] FIG. 2 is a block diagram illustrating an example of the main configuration of a display image generating unit. [Figure 19] 10 is a flowchart illustrating an example of the flow of an imaging process. [Figure 20] 10 is a flowchart illustrating an example of the flow of a display image display process. [Figure 21] 10 is a flowchart illustrating an example of the flow of a display ommatidium selection process. [Figure 22] FIG. 10 is a diagram illustrating an example of a displayed image. [Figure 23] FIG. 10 is a diagram illustrating an example of a displayed image. [Figure 24] 1 is a perspective view showing an example of the configuration of an embodiment of a camera to which the present technology is applied; [Figure 25] FIG. 2 is a block diagram illustrating an example of the electrical configuration of the camera system. [Figure 26] FIG. 2 is a block diagram showing an example of the electrical configuration of the camera. [Figure 27] FIG. 2 is a block diagram illustrating an example of the main configuration of a display image generating unit. [Figure 28] 10 is a flowchart illustrating an example of the flow of a display image display process. [Figure 29] FIG. 2 is a block diagram illustrating an example of the main configuration of a display image generating unit. [Figure 30]10 is a flowchart illustrating an example of the flow of a display image display process. [Figure 31] FIG. 1 is a block diagram illustrating an example of the main configuration of a computer. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, modes for carrying out the present disclosure (hereinafter referred to as embodiments) will be described in the following order. 1. First embodiment (camera) 2. Second embodiment (camera system) 3. Third embodiment (camera) 4. Notes

[0012] <1. First embodiment> <Camera appearance> FIG. 1 is a perspective view showing an example of the configuration of an embodiment of a camera to which the present technology is applied.

[0013] The camera 10 has a built-in image sensor and captures an image of a subject by receiving light rays collected by a lens and performing photoelectric conversion. Hereinafter, an image obtained by such imaging will also be referred to as a captured image.

[0014] Camera 10 is provided with lens barrel 20 on the front side (the side where light is incident) of its image sensor, and lens barrel 20 has a plurality of five unit optical systems 310, 311, 312, 313, and 314. In the following, when it is not necessary to distinguish between unit optical systems 310 to 314, they will be referred to as unit optical system 31 (or unit optical system 31 i ) is called.

[0015] The multiple unitary optical systems 31 are configured so that the optical paths of light passing through them are independent of one another. That is, light that has passed through each unitary optical system 31 of the lens barrel 20 irradiates different positions on the light-receiving surface (e.g., effective pixel area) of the image sensor without entering the other unitary optical systems 31. At least the optical axes of each unitary optical system 31 are located at different positions on the light-receiving surface of the image sensor, and at least a portion of the light that has passed through each unitary optical system 31 irradiates different positions on the light-receiving surface of the image sensor.

[0016] Therefore, in the captured image generated by the image sensor (the entire image output by the image sensor), images of the subject formed through each optical unit 31 are formed at different positions. In other words, captured images (also referred to as viewpoint images) with each optical unit 31 as a viewpoint are obtained from the captured image. That is, the camera 10 can obtain a plurality of viewpoint images by capturing an image of the subject. These multiple viewpoint images can be used, for example, for generating depth information and for processing such as refocusing using the depth information.

[0017] In the following description, an example will be given in which the camera 10 has five optical units 31, but the number of optical units 31 may be any number as long as it is two or more.

[0018] The five unitary optical systems 31 are arranged on a two-dimensional plane perpendicular to the optical axis of the lens barrel 20 (parallel to the light receiving surface (imaging surface) of the image sensor) so that the unitary optical system 310 is at the center (center of gravity) and the other four unitary optical systems 311 to 314 form the vertices of a rectangle. Of course, the arrangement shown in Fig. 1 is just one example, and the positional relationship of each unitary optical system 31 is arbitrary as long as the optical paths are independent of each other.

[0019] Furthermore, the side of camera 10 on which light from a subject enters is referred to as the front. Fig. 2 is a view of camera 10 as seen from the rear. As shown in Fig. 2, a display panel unit 33, a viewfinder unit 34, a dial 35, and buttons 36 are provided on the rear side of the housing of camera 10.

[0020] The display panel unit 33 and the viewfinder unit 34 are made up of display devices such as a liquid crystal display or an organic electroluminescence display (OLED), and can display a through image before capturing an image, or display a captured image for confirmation after capturing an image. The dial 35 and the button 36 are examples of input devices, and can accept user operations when the user operates them.

[0021] <Example of camera electrical configuration> 3 is a block diagram showing an example of the electrical configuration of camera 10. Camera 10 has a multi-eye optical system 30, an image sensor 51, a RAW signal processing unit 52, a region extraction unit 53, a camera signal processing unit 54, a display image generation unit 55, a region identification unit 56, an image reconstruction processing unit 57, a bus 60, a display unit 61, a storage unit 62, a communication unit 64, a filing unit 65, a control unit 81, a storage unit 82, and an optical system control unit 84.

[0022] <Multi-lens optical system> The multi-eye optical system 30 is made up of the above-mentioned unit optical systems 31 (for example, unit optical systems 310 to 314). Each unit optical system 31 of the multi-eye optical system 30 focuses light rays from the subject onto the image sensor 51 of the camera 10. The specifications of each unit optical system 31 are assumed to be identical to one another.

[0023] The unitary optical system 31 has optical elements such as a plurality of lenses arranged in the optical axis direction of the lens barrel, and an aperture which is a mechanism for adjusting the amount of light (F-number) incident on the image sensor 51 through the plurality of lenses by controlling the degree of opening of an obstruction. The unitary optical system 31 may be configured to be able to control the zoom magnification by controlling the position of the lenses.

[0024] <Image sensor> The image sensor 51 is, for example, a CMOS (Complementary Metal Oxide Semiconductor) image sensor, which captures an image of a subject and generates a captured image. Light rays collected by each of the unitary optical systems 310 to 314 are irradiated onto the light receiving surface of the image sensor 51. An image in the captured image corresponding to an area where irradiated light is input to the image sensor 51 via one unitary optical system 31 is also referred to as a unitary image. In other words, the image sensor 51 receives the light rays (irradiated light) and performs photoelectric conversion to generate a captured image including unitary images having each unitary optical system 31 as a viewpoint.

[0025] The optical axes of the unitary optical systems 31 correspond to different positions on the image sensor 51. Therefore, in a captured image (or captured image) generated by the image sensor 51, at least a portion of each unitary image is generated at a different position (not all unitary images are generated at exactly the same position). Furthermore, the unitary images have peripheral portions that are not effective as images. Furthermore, a captured image that includes all unitary images (i.e., the entire captured image generated by the image sensor 51, or an image obtained by removing from the captured image some or all of the areas outside all of the unitary images included in the captured image) is also referred to as a whole image.

[0026] The image sensor 51 may be a single-color (so-called monochrome) image sensor, or may be a color image sensor in which color filters in a Bayer array, for example, are arranged in a pixel group. That is, the captured image output by the image sensor 51 may be a monochrome image or a color image. In the following description, the image sensor 51 will be described as a color image sensor that generates and outputs a captured image in RAW format.

[0027] In this embodiment, the RAW format refers to an image in a state in which the positional relationship of the arrangement of the color filters of the image sensor 51 is maintained, and may include an image that has been subjected to signal processing and compression encoding, such as image size conversion processing, noise reduction processing, and defect correction processing of the image sensor 51, for the image output from the image sensor 51. Furthermore, a captured image in the RAW format does not include a monochrome image.

[0028] The image sensor 51 can output a captured image (whole image) in RAW format generated by photoelectrically converting irradiated light. For example, the image sensor 51 can supply the captured image (whole image) in RAW format to at least one of the bus 60, the RAW signal processing unit 52, the area extraction unit 53, and the area identification unit 56.

[0029] For example, the image sensor 51 can supply the captured image (whole image) in the RAW format to the storage unit 62 via the bus 60, and store it in the storage medium 63. The image sensor 51 can also supply the captured image (whole image) in the RAW format to the communication unit 64 via the bus 60, and transmit it to the outside of the camera 10. The image sensor 51 can also supply the captured image (whole image) in the RAW format to the filing unit 65 via the bus 60, and file it. The image sensor 51 can also supply the captured image (whole image) in the RAW format to the image reconstruction processing unit 57 via the bus 60, and have it perform image reconstruction processing.

[0030] The image sensor 51 may be a single-plate image sensor, or may be a set of image sensors (also called a multi-plate image sensor) including a plurality of image sensors, such as a three-plate image sensor.

[0031] For example, as a three-panel image sensor, there is one that has three image sensors (image sensors 51-1 to 51-3) for each of RGB (Red, Green, Blue) as shown in FIG. 4. In this case, the light rays from the subject are separated for each wavelength range using an optical system (optical path separation unit) such as a prism and are incident on each image sensor. The image sensors 51-1 to 51-3 photoelectrically convert the incident light respectively. That is, the image sensors 51-1 to 51-3 photoelectrically convert light in mutually different wavelength ranges at substantially the same timing. Therefore, in the case of a multi-panel image sensor, in each image sensor, captured images (that is, images having substantially the same pattern except for different wavelength ranges) captured at substantially the same angle of view at substantially the same time are obtained. Therefore, the position and size of the viewpoint image region (described later) in the captured images obtained by each image sensor are substantially the same as each other. In this case, the combined R image, G image, and B image can be regarded as a captured image in RAW format.

[0032] Note that in the case of a multi-panel image sensor, each image sensor is not limited to those for each of RGB and may all be monochrome, or may all be equipped with a color filter such as a Bayer array. When all are color filters such as a Bayer array, if all the arrays are the same and the positional relationship of the pixels with each other is adjusted, for example, noise reduction can be performed. If the positional relationship of each RGB image sensor is shifted, it is also possible to improve the image quality by using the effect of spatial pixel shift.

[0033] In the case of such a multi-panel imaging device as well, a plurality of individual-eye images and a plurality of viewpoint images are included in the captured image output from each image sensor, that is, one image sensor.

[0034] <RAW signal processing unit> The RAW signal processing unit 52 performs signal processing on a RAW format image. For example, the RAW signal processing unit 52 can acquire a captured image (whole image) in RAW format supplied from the image sensor 51. The RAW signal processing unit 52 can also perform predetermined signal processing on the acquired captured image. The content of this signal processing is arbitrary. For example, it may be defect correction, noise reduction, compression (encoding), or other signal processing. Of course, the RAW signal processing unit 52 can also perform multiple signal processing on the captured image. Note that various signal processing on a RAW format image is limited to an image after signal processing in which the positional relationship of the color filters of the image sensor 51 is maintained as described above (in the case of a multi-chip imaging device, an image in the form of R, G, and B images).

[0035] The RAW signal processing unit 52 can supply the signal-processed captured image (RAW') in RAW format or the compressed (encoded) captured image (compressed RAW) to the storage unit 62 via the bus 60, and store it in the storage medium 63. The RAW signal processing unit 52 can also supply the signal-processed captured image (RAW') in RAW format or the compressed (encoded) captured image (compressed RAW) to the communication unit 64 via the bus 60, and transmit it. The RAW signal processing unit 52 can also supply the signal-processed captured image (RAW') in RAW format or the compressed (encoded) captured image (compressed RAW) to the filing unit 65 via the bus 60, and file it. The RAW signal processing unit 52 can also supply the signal-processed captured image (RAW') in RAW format or the compressed (encoded) captured image (compressed RAW) to the image reconstruction processing unit 57 via the bus 60, and cause the image reconstruction processing unit 57 to perform image reconstruction processing. When there is no need to distinguish between RAW, RAW', and compressed RAW (all shown in FIG. 3), they will be referred to as RAW images.

[0036] <Area extraction part> The area extraction unit 53 performs processing related to extraction of a partial area from a captured image in RAW format (cutting out a partial image). For example, the area extraction unit 53 can acquire a captured image in RAW format (whole image) supplied from the image sensor 51. The area extraction unit 53 can also acquire information indicating an area to be extracted from the captured image (also referred to as extracted area information) supplied from the area identification unit 56. Then, the area extraction unit 53 can extract a partial area from the captured image (cut out a partial image) based on the extracted area information.

[0037] For example, the region extraction unit 53 can extract an image having each unitary optical system 31 as a viewpoint from the captured image (whole image). That is, the region extraction unit 53 can extract an effective portion from the region of each unitary image included in the captured image as an image having each unitary optical system 31 as a viewpoint. The extracted effective portion (a part of the unitary image) is also referred to as a viewpoint image. The extracted region in the captured image (a region corresponding to the viewpoint image) is also referred to as a viewpoint image region. For example, the region extraction unit 53 can acquire, as extraction region information, viewpoint-related information used to identify the viewpoint image region, supplied from the region identification unit 56, and extract each viewpoint image region indicated in the viewpoint-related information from the captured image (cut out each viewpoint image). Then, the region extraction unit 53 can supply each extracted viewpoint image (RAW format) to the camera signal processing unit 54.

[0038] Furthermore, for example, the area extraction unit 53 can synthesize each viewpoint image cut out from the captured image (whole image) to generate a composite image. The composite image is obtained by synthesizing each viewpoint image and converting it into a single piece of data or a single image. For example, the area extraction unit 53 can generate a single image (composite image) in which each viewpoint image is arranged in a plane. The area extraction unit 53 can supply the generated composite image (RAW format) to the camera signal processing unit 54.

[0039] Furthermore, for example, the region extraction unit 53 can supply the entire image to the camera signal processing unit 54. For example, the region extraction unit 53 can extract a partial region including all unitary images from the acquired captured image (that is, cut out a partial image including all unitary images), and supply the cut-out partial image (that is, an image in which part or all of the region outside all of the unitary images included in the captured image has been deleted) as an entire image in RAW format to the camera signal processing unit 54. In this case, the location (range) of the region to be extracted may be predetermined by the region extraction unit 53, or may be specified by viewpoint-related information supplied from the region identification unit 56.

[0040] The area extraction unit 53 can also supply the acquired captured image (i.e., the entire captured image, rather than a partial image including all the cut-out ommatidial images) to the camera signal processing unit 54 as a whole image in RAW format.

[0041] In addition, the area extraction unit 53 can supply the partial image in RAW format (whole image, viewpoint image, or composite image) cut out from the captured image as described above to the memory unit 62, communication unit 64, filing unit 65, image reconstruction processing unit 57, etc. via the bus 60, as in the case of the image sensor 51.

[0042] The region extraction unit 53 can also supply the partial image (whole image, viewpoint image, or composite image) in the RAW format to the RAW signal processing unit 52, which can then perform predetermined signal processing or compress (encode) the image. In this case, the RAW signal processing unit 52 can also supply the captured image in the RAW format (RAW') that has been subjected to signal processing or the compressed (encoded) captured image (compressed RAW) via the bus 60 to the storage unit 62, the communication unit 64, the filing unit 65, the image reconstruction processing unit 57, or the like.

[0043] That is, at least one of the captured image (or the entire image), the viewpoint image, and the composite image may be a RAW image.

[0044] <Camera signal processing section> The camera signal processing unit 54 performs camera signal processing on an image. For example, the camera signal processing unit 54 can acquire an image (whole image, viewpoint image, or composite image) supplied from the region extraction unit 53. The camera signal processing unit 54 can also perform camera signal processing (camera process) on the acquired image. For example, the camera signal processing unit 54 can perform color separation processing (demosaic processing when a mosaic color filter such as a Bayer array is used) on the image to be processed, which separates each of the RGB colors to generate an R image, a G image, and a B image each having the same number of pixels as the image to be processed, and YC conversion processing, which converts the color space of the color-separated image from RGB to YC (luminance and color difference). The camera signal processing unit 54 can also perform processes such as defect correction, noise reduction, AWB (Automatic White Balance), and gamma correction on the image to be processed. Furthermore, the camera signal processing unit 54 can compress (encode) the image to be processed. Of course, the camera signal processing unit 54 can also perform multiple camera signal processing operations on the image to be processed, and can also perform camera signal processing other than the examples described above.

[0045] In the following description, it is assumed that the camera signal processing unit 54 acquires an image in RAW format, performs color separation processing and YC conversion on the image, and outputs an image (YC) in YC format. This image may be an entire image, an image from each viewpoint, or a composite image. Furthermore, this image (YC) in YC format may or may not be coded. In other words, the data output from the camera signal processing unit 54 may be coded data or may be uncoded image data.

[0046] That is, at least one of the captured image (or the entire image), the viewpoint image, and the composite image may be an image in the YC format (also referred to as a YC image).

[0047] Furthermore, the image output by the camera signal processing unit 54 may not be fully developed, but may be an YC format image (YC) that has not undergone some or all of the processing related to irreversible image quality adjustment (color adjustment) such as gamma correction or color matrix. In this case, the YC format image (YC) can be restored to a RAW format image with almost no degradation in a later stage or during playback.

[0048] The camera signal processing unit 54 can, for example, supply the YC format image (YC) that has been subjected to the camera signal processing to a display unit 61 via the bus 60 and display it. The camera signal processing unit 54 can also supply the YC format image (YC) that has been subjected to the camera signal processing to a storage unit 62 via the bus 60 and store it in a storage medium 63. The camera signal processing unit 54 can also supply the YC format image (YC) that has been subjected to the camera signal processing to a communication unit 64 via the bus 60 and transmit it to the outside. The camera signal processing unit 54 can also supply the YC format image (YC) that has been subjected to the camera signal processing to a filing unit 65 via the bus 60 and file it. The camera signal processing unit 54 can also supply the YC format image (YC) that has been subjected to the camera signal processing to an image reconstruction processing unit 57 via the bus 60 and have it perform image reconstruction processing.

[0049] Furthermore, for example, the camera signal processing unit 54 can also supply the YC format image (YC) to the display image generating unit 55.

[0050] Note that, when a RAW format image (whole image, viewpoint image, or partial image) is stored in the storage medium 63, the camera signal processing unit 54 may be configured to read out the RAW format image from the storage medium 63 and perform signal processing on it. In this case, too, the camera signal processing unit 54 can supply the YC format image (YC) that has been subjected to camera signal processing via the bus 60 to the display unit 61, the storage unit 62, the communication unit 64, the filing unit 65, the image reconstruction processing unit 57, or the like.

[0051] In addition, the camera signal processing unit 54 may perform camera signal processing on the captured image (whole image) in RAW format output from the image sensor 51, and the area extraction unit 53 may extract a partial area from the captured image (whole image) after the camera signal processing.

[0052] <Display image generation unit> The display image generation unit 55 performs processing related to the generation of a display image to be displayed on the display unit 61. For example, the display image generation unit 55 can generate a through image. A through image is an image that is displayed during shooting or during shooting preparation (non-recording) so that the user can check an image being prepared for shooting. In other words, a through image is generated using a display image (also referred to as a captured image) generated by the image sensor 51. Note that a through image is also referred to as a live view image or an EE (Electronic to Electronic) image. Note that, during still image shooting, a through image is an image before shooting, but during video shooting, a through image corresponding to not only an image being prepared for shooting but also an image being shot (recorded) is displayed.

[0053] A captured image is an image other than a captured image generated by the image sensor 51. In other words, a captured image can be generated at a timing other than the timing at which a captured image is generated. Similarly to a captured image, a captured image is an image generated by photoelectrically converting light received by the image sensor 51. However, the use of a captured image differs from the use of a captured image. While a captured image is an image for recording (storage), a captured image is an image to be displayed as a through image or the like on the display unit 61 (display panel unit 33 or viewfinder unit 34). Furthermore, while a captured image is a still image (or may be a moving image), a captured image (through image) is basically displayed as a moving image (an image consisting of multiple frames). The specifications of a captured image (e.g., resolution, aspect ratio, color, etc.) are arbitrary and may be the same as or different from the captured image. For example, a captured image may have a lower resolution than a captured image.

[0054] As with a captured image, this captured image may have a partial region extracted by the region extraction unit 53, or may undergo camera signal processing by the camera signal processing unit 54. That is, as with a captured image, the captured image may be supplied to the display image generation unit 55 as, for example, an image in YC format (whole image, viewpoint image, or composite image).

[0055] In this case, the display image generating unit 55 can acquire a captured image (for example, an entire image, a viewpoint image, or a composite image) supplied from the camera signal processing unit 54. The display image generating unit 55 can generate a display image by using the acquired captured image to perform image size (resolution) conversion, for example, to convert the image into an image size according to the resolution of the display unit 61. The display image generating unit 55 can then supply the generated display image to the display unit 61 via the bus 60 and display it as a through image. The specifications of this display image are arbitrary and may be the same as or different from the captured image.

[0056] Furthermore, for example, the display image generation unit 55 can also generate a confirmation image of the captured image. For example, when a user issues an instruction to capture an image by pressing the shutter button, the camera 10 captures an image of a subject using the image sensor 51 or the like and generates a captured image for recording (storage). At that time, the camera 10 generates a confirmation image (i.e., a display image) using the captured image so that the user can confirm the generated captured image (i.e., the imaging result), and displays the image on the display unit 61 (display panel unit 33 or viewfinder unit 34). The display image generation unit 55 can generate this confirmation image.

[0057] That is, in this case, the display image generating unit 55 can acquire a captured image (for example, an entire image, a viewpoint image, or a composite image) supplied from, for example, the camera signal processing unit 54. The display image generating unit 55 can generate a display image using the acquired captured image. Furthermore, the display image generating unit 55 can supply the generated display image to the display unit 61 via the bus 60 and display it as a confirmation image. Note that the specifications of this confirmation image are arbitrary, as in the case of the display image described above.

[0058] Furthermore, for example, the display image generating unit 55 can also generate a display image of a captured image that has already been recorded (saved). For example, the camera 10 can display a captured image that has been read from the storage medium 63 or obtained from another device via the communication unit 64 on the display unit 61. The display image generating unit 55 can generate a display image of this captured image.

[0059] That is, in this case, the display image generating unit 55 can acquire captured images read from the storage medium 63 by the storage unit 62, or captured images (for example, an entire image, a viewpoint image, or a composite image) acquired from another device via the communication unit 64. The display image generating unit 55 can generate a display image using the acquired captured images. Furthermore, the display image generating unit 55 can supply the generated display image to the display unit 61 via the bus 60 for display. The specifications of this display image are arbitrary.

[0060] The display image generating unit 55 can acquire viewpoint-related information (VI or VI') supplied from the area specifying unit 56 and generate these display images using the acquired viewpoint-related information. The display image generating unit 55 can also acquire control information supplied from the control unit 81 and generate these display images using the control information.

[0061] <Area identification part> The region specifying unit 56 performs processing related to specifying (setting) the region to be extracted from the captured image by the region extraction unit 53. For example, the region specifying unit 56 specifies viewpoint-related information (VI) and supplies the viewpoint image region to the region extraction unit 53.

[0062] The viewpoint-related information (VI) includes, for example, viewpoint area information indicating a viewpoint image area in the captured image. The viewpoint area information may represent the viewpoint image area in any way. For example, the viewpoint image area may be represented by coordinates indicating a position corresponding to the optical axis of the single optical system 31 in the captured image (also referred to as the center coordinates of the viewpoint image area) and the resolution (number of pixels) of the viewpoint image (viewpoint image area). In other words, the viewpoint area information may include the center coordinates of the viewpoint image area in the captured image and the resolution of the viewpoint image area. In this case, the location of the viewpoint image area within the entire image can be identified from the center coordinates of the viewpoint image area and the resolution (number of pixels) of the viewpoint image area.

[0063] Note that the viewpoint area information is set for each viewpoint image area. In other words, when a captured image includes multiple viewpoint images, the viewpoint-related information (VI) may include viewpoint identification information (e.g., an identification number) for identifying the viewpoint image (area) and viewpoint area information for each viewpoint image (each viewpoint image area).

[0064] The viewpoint-related information (VI) may also include any other information. For example, the viewpoint-related information (VI) may include viewpoint time information indicating the time when a captured image from which a viewpoint image is extracted was captured. The viewpoint-related information (VI) may also include viewpoint image-containing area information indicating a viewpoint image-containing area that is an area cut out from a unitary image and that contains the viewpoint image area. Furthermore, the viewpoint-related information (VI) may also include spot light information (SI) that is information regarding an image of a spot light formed in an area that is neither the viewpoint image area of ​​the captured image nor the unitary image area.

[0065] The area identification unit 56 supplies such viewpoint-related information (VI) to the area extraction unit 53 as information indicating the identified viewpoint image area, and the area extraction unit 53 can extract the viewpoint image area identified by the area identification unit 56 (cut out the viewpoint image) based on the viewpoint-related information (VI).

[0066] The area specifying unit 56 can also supply the viewpoint-related information (VI) to the bus 60. For example, the area specifying unit 56 can supply the viewpoint-related information (VI) to the storage unit 62 via the bus 60 and store it in a storage medium 63. The area specifying unit 56 can also supply the viewpoint-related information (VI) to the communication unit 64 via the bus 60 and cause it to be transmitted. The area specifying unit 56 can also supply the viewpoint-related information (VI) to the filing unit 65 via the bus 60 and cause it to be filed. The area specifying unit 56 can also supply the viewpoint-related information (VI) to the image reconstruction processing unit 57 via the bus 60 and cause it to be used in image reconstruction processing.

[0067] For example, the area specifying unit 56 may acquire such viewpoint-related information (VI) from the control unit 81 and supply the acquired viewpoint-related information (VI) to the area extraction unit 53 and the bus 60. In this case, the control unit 81 reads out the viewpoint-related information (VI) stored in the storage medium 83 via the storage unit 82 and supplies it to the area specifying unit 56. The area specifying unit 56 supplies the viewpoint-related information (VI) to the area extraction unit 53 and the bus 60. Note that this viewpoint-related information (VI) may include spot light information (SI).

[0068] The viewpoint-related information (VI) supplied to the storage unit 62, the communication unit 64, or the filing unit 65 via the bus 60 in this manner is associated therewith with an image (an entire image, a viewpoint image, or a composite image). For example, the storage unit 62 can associate the supplied viewpoint-related information (VI) with an image (an entire image, a viewpoint image, or a composite image) and store it in the storage medium 63. The communication unit 64 can associate the supplied viewpoint-related information (VI) with an image (an entire image, a viewpoint image, or a composite image) and transmit it to the outside. Furthermore, the filing unit 65 can associate the supplied viewpoint-related information (VI) with an image (an entire image, a viewpoint image, or a composite image) and generate a file including them.

[0069] Furthermore, the region identification unit 56 may acquire a captured image in RAW format supplied from the image sensor 51, generate viewpoint-related information (VI') based on the captured image, and supply the generated viewpoint-related information (VI') to the region extraction unit 53 and the bus 60. In this case, the region identification unit 56 identifies each viewpoint image region from the captured image and generates viewpoint-related information (VI') indicating the viewpoint image region (for example, indicating the viewpoint image region by the center coordinates of the viewpoint image region in the captured image, the resolution of the viewpoint image region, etc.). Then, the region identification unit 56 supplies the generated viewpoint-related information (VI') to the region extraction unit 53 and the bus 60. Note that the viewpoint-related information (VI') may include spot light information (SI') generated by the region identification unit 56 based on the captured image.

[0070] Furthermore, the region identification unit 56 may acquire the viewpoint-related information (VI) from the control unit 81, acquire the captured image in RAW format supplied from the image sensor 51, generate spot light information (SI') based on the captured image, add the spot light information (SI') to the viewpoint-related information (VI), and supply it to the region extraction unit 53 and the bus 60. In this case, the control unit 81 reads out the viewpoint-related information (VI) stored in the storage medium 83 via the storage unit 82 and supplies it to the region identification unit 56. The region identification unit 56 adds the spot light information (SI') to the viewpoint-related information (VI) to generate the viewpoint-related information (VI'). The region identification unit 56 supplies the viewpoint-related information (VI') to the region extraction unit 53 and the bus 60.

[0071] Alternatively, the region identification unit 56 may acquire the viewpoint-related information (VI) from the control unit 81, acquire a captured image in RAW format supplied from the image sensor 51, generate spot light information (SI') based on the captured image, correct the viewpoint-related information (VI) using the spot light information (SI'), and supply the corrected viewpoint-related information (VI') to the region extraction unit 53 or the bus 60. In this case, the control unit 81 reads out the viewpoint-related information (VI) stored in the storage medium 83 via the storage unit 82 and supplies it to the region identification unit 56. The region identification unit 56 corrects the viewpoint-related information (VI) using the spot light information (SI') to generate the viewpoint-related information (VI'). The region identification unit 56 supplies the viewpoint-related information (VI') to the region extraction unit 53 and the bus 60.

[0072] The region specifying unit 56 can also supply the viewpoint-related information (VI or VI′) to the display image generating unit 55.

[0073] <Image reconstruction processing unit> The image reconstruction processing unit 57 performs processing related to image reconstruction. For example, the image reconstruction processing unit 57 can acquire YC format images (whole images, viewpoint images, or composite images) from the camera signal processing unit 54 or the storage unit 62 via the bus 60. The image reconstruction processing unit 57 can also acquire viewpoint-related information from the area identification unit 56 or the storage unit 62 via the bus 60.

[0074] Furthermore, the image reconstruction processing unit 57 can perform image processing such as generating depth information and refocusing to generate (reconstruct) an image focused on an arbitrary subject, using the acquired image and viewpoint-related information associated with the acquired image. For example, when viewpoint images are the processing target, the image reconstruction processing unit 57 performs processing such as generating depth information and refocusing using each viewpoint image. Furthermore, when a captured image or a composite image is the processing target, the image reconstruction processing unit 57 extracts each viewpoint image from the captured image or composite image, and performs processing such as generating depth information and refocusing using the extracted viewpoint image.

[0075] The image reconstruction processing unit 57 can supply the generated depth information and refocused images as processing results to a storage unit 62 via a bus 60, and store them externally in a storage medium 63. The image reconstruction processing unit 57 can also supply the generated depth information and refocused images as processing results to a communication unit 64 via the bus 60, and transmit them. The image reconstruction processing unit 57 can also supply the generated depth information and refocused images as processing results to a filing unit 65 via the bus 60, and file them.

[0076] <Bus> The image sensor 51, RAW signal processing unit 52, area extraction unit 53, camera signal processing unit 54, display image generation unit 55, area identification unit 56, image reconstruction processing unit 57, display unit 61, storage unit 62, communication unit 64, and filing unit 65 are connected to the bus 60. The bus 60 functions as a transmission medium (transmission path) for various data exchanged between these blocks. Note that the bus 60 may be realized by wired or wireless communication.

[0077] <Display section> The display unit 61 includes, for example, a display panel unit 33 and a viewfinder unit 34. The display unit 61 performs processing related to image display. For example, the display unit 61 can acquire a display image of a captured image in YC format supplied from the display image generation unit 55, convert it to RGB format, and display it as a through image on the display panel unit 33 or the viewfinder unit 34. In addition, the display unit 61 can also display information such as menus and settings of the camera 10.

[0078] The display unit 61 can also acquire a confirmation image of the captured image supplied from the display image generation unit 55 and display it on the display panel unit 33 or the viewfinder unit 34. The display unit 61 can also acquire a display image of the recorded (saved) captured image supplied from the display image generation unit 55 and display it on the display panel unit 33 or the viewfinder unit 34. The display unit 61 may also be configured to display a thumbnail image of the captured image.

[0079] <Storage section> The storage unit 62 controls the storage of a storage medium 63, which may be, for example, a semiconductor memory. The storage medium 63 may be a removable storage medium, or may be a storage medium built into the camera 10.

[0080] The storage unit 62 can store images (captured images, viewpoint images, or composite images) supplied via the bus 60 in the storage medium 63 in response to operations by the control unit 81 or the user.

[0081] For example, the storage unit 62 can acquire a RAW format image (whole image, viewpoint image, or composite image) supplied from the image sensor 51 or the area extraction unit 53 and store it in the storage medium 63. The storage unit 62 can also acquire a RAW format image (whole image, viewpoint image, or composite image) that has been subjected to signal processing and that is supplied from the RAW signal processing unit 52 and store it in the storage medium 63. The storage unit 62 can also acquire a compressed (encoded) RAW format image (whole image, viewpoint image, or composite image) that is supplied from the RAW signal processing unit 52 and store it in the storage medium 63. The storage unit 62 can also acquire a YC format image (whole image, viewpoint image, or composite image) that is supplied from the camera signal processing unit 54 and store it in the storage medium 63.

[0082] At this time, the image (whole image, viewpoint image, or composite image) and the viewpoint-related information can be associated with each other and stored in the storage medium 63. For example, the storage unit 62 can acquire the viewpoint-related information supplied from the area identification unit 56, associate it with the image (whole image, viewpoint image, or composite image), and store it in the storage medium 63. In other words, the storage unit 62 functions as an associating unit that associates at least one of the whole image, viewpoint image, and composite image with the viewpoint-related information.

[0083] Additionally, the storage unit 62 can acquire depth information and refocused images supplied from the image reconstruction processing unit 57 and store them in the storage medium 63. The storage unit 62 can also acquire files supplied from the filing unit 65 and store them in the storage medium 63. These files include images (whole images, viewpoint images, or composite images) and viewpoint-related information. That is, in these files, the images (whole images, viewpoint images, or composite images) and viewpoint-related information are associated with each other.

[0084] Furthermore, the storage unit 62 can read out data, files, etc. stored in the storage medium 63 in response to operations by the control unit 81 or the user.

[0085] For example, the storage unit 62 can read out a captured image (whole image, viewpoint image, or composite image) in YC format from the storage medium 63, supply it to the display image generation unit 55, and cause the display image to be generated. The storage unit 62 can also read out an image (whole image, viewpoint image, or composite image) in RAW format from the storage medium 63, supply it to the camera signal processing unit 54, and cause the camera signal processing to be performed on it.

[0086] Furthermore, the storage unit 62 can read out, together with the image (whole image, viewpoint image, or composite image), viewpoint-related information associated with the image. For example, the storage unit 62 can read out data of the images (whole image, viewpoint image, or composite image) and viewpoint-related information associated with each other from the storage medium 63, and supply them to the image reconstruction processing unit 57 to perform processing such as generating depth information and refocusing. The storage unit 62 can also read out a file containing the images (whole image, viewpoint image, or composite image) and viewpoint-related information associated with each other from the storage medium 63, and supply the file to the image reconstruction processing unit 57 to perform processing such as generating depth information and refocusing.

[0087] Furthermore, the storage unit 62 can read out data and files of images (whole image, viewpoint image, or composite image) and viewpoint-related information that are associated with each other from the storage medium 63, and supply them to the communication unit 64 for transmission. The storage unit 62 can also read out data of images (whole image, viewpoint image, or composite image) and viewpoint-related information that are associated with each other from the storage medium 63, and supply them to the filing unit 65 for filing.

[0088] The storage medium 63 may be a ROM (Read Only Memory) or a rewritable memory such as a RAM (Random Access Memory) or a flash memory. In the case of a rewritable memory, the storage medium 63 can store any information.

[0089] <Communications Department> The communication unit 64 communicates with a server on the Internet, a PC on a wired or wireless LAN, other external devices, etc., using any communication method. In response to the control of the control unit 81 or the user's operation, the communication unit 64 can transmit data and files such as images (captured images, viewpoint images, or composite images) and viewpoint-related information to the communication partner (external device) using a streaming method, an upload method, etc.

[0090] For example, the communication unit 64 can acquire and transmit a RAW format image (captured image, viewpoint image, or composite image) supplied from the image sensor 51 or the area extraction unit 53. The communication unit 64 can also acquire and transmit a RAW format image (captured image, viewpoint image, or composite image) that has been subjected to signal processing and that is supplied from the RAW signal processing unit 52, or a compressed (encoded) image (captured image, viewpoint image, or composite image). The communication unit 64 can also acquire and transmit a YC format image (captured image, viewpoint image, or composite image) supplied from the camera signal processing unit 54.

[0091] In this case, the communication unit 64 can acquire the viewpoint-related information supplied from the region identification unit 56 and associate it with the above-mentioned image (the entire image, the viewpoint image, or the composite image). That is, the communication unit 64 can transmit the image (the entire image, the viewpoint image, or the composite image) and the viewpoint-related information in association with each other. For example, when transmitting an image by streaming, the communication unit 64 acquires the image to be transmitted (the entire image, the viewpoint image, or the composite image) from the processing unit that supplies the image, and repeats the process of associating the image with the viewpoint-related information supplied from the region identification unit 56 and transmitting it. That is, the communication unit 64 functions as an associating unit that associates at least one of the entire image, the viewpoint image, and the composite image with the viewpoint-related information.

[0092] Furthermore, for example, the communication unit 64 can acquire and transmit depth information and refocused images supplied from the image reconstruction processing unit 57. Furthermore, the communication unit 64 can acquire and transmit files supplied from the filing unit 65. These files include, for example, images (whole images, viewpoint images, or composite images) and viewpoint-related information. That is, in these files, the images (whole images, viewpoint images, or composite images) and viewpoint-related information are associated with each other.

[0093] The communication unit 64 can also acquire data and files of images (whole images, viewpoint images, or composite images) and viewpoint-related information from devices (communication partners) external to the camera 10.

[0094] <Filing section> The filing unit 65 performs processing related to file generation. For example, the filing unit 65 can acquire a RAW format image (whole image, viewpoint image, or composite image) supplied from the image sensor 51 or the area extraction unit 53. The filing unit 65 can also acquire a RAW format image (whole image, viewpoint image, or composite image) that has been subjected to signal processing supplied from the RAW signal processing unit 52, or a compressed (encoded) RAW format image (whole image, viewpoint image, or composite image). The filing unit 65 can also acquire a YC format image (whole image, viewpoint image, or composite image) supplied from the camera signal processing unit 54. For example, the filing unit 65 can also acquire viewpoint-related information supplied from the area identification unit 56.

[0095] The filing unit 65 can associate the acquired multiple pieces of data with each other by filing the multiple pieces of data and generating a single file containing the multiple pieces of data. For example, the filing unit 65 can associate the above-mentioned images (whole image, viewpoint image, or composite image) and viewpoint-related information with each other by generating a single file from them. In other words, the filing unit 65 functions as an associating unit that associates at least one of the whole image, viewpoint image, and composite image with viewpoint-related information.

[0096] Also, for example, the filing unit 65 can acquire and file depth information and refocused images supplied from the image reconstruction processing unit 57. Furthermore, the filing unit 65 can generate one file from images (whole image, viewpoint image, or composite image) and viewpoint-related information that are associated with each other and supplied from the storage unit 62.

[0097] The filing unit 65 can generate a thumbnail image of the image to be filed (for example, a viewpoint image) and include it in the generated file. That is, by filing, the filing unit 65 can associate this thumbnail image with the image (whole image, viewpoint image, or composite image) and viewpoint-related information.

[0098] The filing unit 65 can supply the generated file (the images and viewpoint-related information associated with each other) to the storage unit 62 via the bus 60, for example, and store it in the storage medium 63. The filing unit 65 can also supply the generated file (the images and viewpoint-related information associated with each other) to the communication unit 64 via the bus 60, for example, and transmit it.

[0099] <Association section> The storage unit 62, communication unit 64, and filing unit 65 are also referred to as an associating unit 70. The associating unit 70 associates an image (an entire image, a viewpoint image, or a composite image) with viewpoint-related information. For example, the storage unit 62 can associate at least one of the entire image, the viewpoint image, and the composite image with the viewpoint-related information and store them in the storage medium 63. The communication unit 64 can associate at least one of the entire image, the viewpoint image, and the composite image with the viewpoint-related information and transmit them. Furthermore, the filing unit 65 can associate at least one of the entire image, the viewpoint image, and the composite image with the viewpoint-related information by generating a file from the image.

[0100] <Control unit> The control unit 81 performs control processing related to the camera 10. That is, the control unit 81 controls each unit of the camera 10 to execute processing. For example, the control unit 81 can control the multi-eye optical system 30 (each unit optical system 31) via the optical system control unit 84 to set the optical system related to imaging, such as the aperture and focus position. The control unit 81 also controls the image sensor 51 to cause the image sensor 51 to perform imaging (photoelectric conversion) and generate a captured image.

[0101] Furthermore, the control unit 81 can supply the viewpoint-related information (VI) to the area specifying unit 56 to specify an area to be extracted from the captured image. Note that the viewpoint-related information (VI) may include spot light information (SI). The control unit 81 may also read out the viewpoint-related information (VI) stored in the storage medium 83 via the storage unit 82 and supply it to the area specifying unit 56.

[0102] The control unit 81 may also supply control information related to the generation of display images to the display image generation unit 55. For example, the control unit 81 may accept a user operation via an input device such as the dial 35 or the button 36, generate control information in accordance with the accepted operation, and supply the control information to the display image generation unit 55. The control unit 81 may also acquire control information read by the storage unit 82 from the storage medium 83, and supply the control information to the display image generation unit 55. The control unit 81 may also acquire control information read by the storage unit 82 from the storage medium 83, accept a user operation via an input device such as the dial 35 or the button 36, update the control information in accordance with the accepted operation, and supply the control information to the display image generation unit 55.

[0103] The control unit 81 can also acquire an image via the bus 60 and control the aperture based on the brightness of the image via the optical system control unit 84. The control unit 81 can also control the focus based on the sharpness of the image via the optical system control unit 84. The control unit 81 can also control the camera signal processing unit 54 based on the RGB ratio of the image to control the white balance gain.

[0104] <Storage section> The storage unit 82 controls storage of a storage medium 83 made of, for example, a semiconductor memory. This storage medium 83 may be a removable storage medium or an internal memory. This storage medium 83 stores, for example, viewpoint-related information (VI). This viewpoint-related information (VI) is information corresponding to (each unit optical system 31 of) the multi-eye optical system 30 and the image sensor 51. In other words, the viewpoint-related information (VI) is information related to a viewpoint image having each unit optical system 31 of the multi-eye optical system 30 as a viewpoint, and is information used to identify the viewpoint image area. For example, this viewpoint-related information (VI) may include spot light information (SI).

[0105] For example, the storage unit 82 can read out the viewpoint-related information (VI) stored in the storage medium 83 in response to a request from the control unit 81 or an operation by the user, and supply it to the control unit 81.

[0106] The storage medium 83 may be a ROM, or may be a rewritable memory such as a RAM or a flash memory. In the case of a rewritable memory, the storage medium 83 can store any information.

[0107] Alternatively, the storage medium 83 may store control information relating to the generation of a display image, and the storage unit 82 may read out the control information in response to a request from the control unit 81 or an operation by the user, and supply the control information to the control unit 81.

[0108] Furthermore, the storage unit 82 and the storage medium 83 may be substituted by the storage unit 62 and the storage medium 63. In other words, the information (such as viewpoint-related information (VI)) to be stored in the storage medium 83 described above may be stored in the storage medium 63. In that case, the storage unit 82 and the storage medium 83 may be omitted.

[0109] <Optical system control unit> The optical system control unit 84 controls the multi-eye optical system 30 (each of the unitary optical systems 31) under the control of the control unit 81. For example, the optical system control unit 84 can control the lens group and aperture of each unitary optical system 31, and can control the focal length or F-number, or both, of each unitary optical system 31. If the camera 10 has an electric focus adjustment function, the optical system control unit 84 can control the focus (focal length) of the multi-eye optical system 30 (of each unitary optical system 31). The optical system control unit 84 may also be configured to control the aperture (F-number) of each unitary optical system 31.

[0110] Instead of providing such an electric focus adjustment function, camera 10 may be provided with a mechanism (physical configuration) for adjusting the focal length by manually operating a focus ring provided on the lens barrel, in which case optical system control unit 84 can be omitted.

[0111] <Associating viewpoint-related information> The camera 10 can extract viewpoint images from a captured image, with each single optical system 31 serving as a viewpoint. Since the multiple viewpoint images extracted from one captured image are images from different viewpoints, these viewpoint images can be used to perform processing such as depth estimation by multi-lens matching and correction for suppressing installation errors in the multi-lens system. However, to perform these processing operations, information such as the relative positions between the viewpoint images is required.

[0112] Therefore, the camera 10 associates viewpoint-related information, which is information used to identify the areas of a plurality of viewpoint images in a captured image, with the entire image, viewpoint image, or composite image that is output.

[0113] Here, the term "associate" means, for example, making it possible to use (link) one piece of data when processing the other piece of data. In other words, the captured image and the viewpoint-related information may be in any form as data (files). For example, the captured image and the viewpoint-related information may be combined into one piece of data (file), or may be separate pieces of data (files). For example, the viewpoint-related information associated with a captured image may be transmitted over a transmission path different from that of the captured image. Furthermore, for example, the viewpoint-related information associated with a captured image may be recorded on a recording medium different from that of the captured image (or on a different recording area of ​​the same recording medium). Of course, the captured image and the viewpoint-related information may be combined into one stream of data or one file.

[0114] The image to which the viewpoint-related information is associated may be a still image or a moving image. In the case of a moving image, area extraction and association of viewpoint-related information can be performed on each frame image in the same way as in the case of a still image.

[0115] Furthermore, this "association" may be with a part of the data (file) rather than with the entire data. For example, if the captured image is a moving image consisting of multiple frames, the viewpoint-related information may be associated with any unit of the captured image, such as multiple frames, one frame, or a part of a frame.

[0116] When the captured image and the viewpoint-related information are stored as separate data (files), the two can be associated by assigning the same identification number to both the captured image and the viewpoint-related information. When the captured image and the viewpoint-related information are combined into one file, the viewpoint-related information may be added to the header of the captured image, for example. The object to which the viewpoint-related information is associated may be the captured image (whole image), a viewpoint image, or a composite image of viewpoint images.

[0117] <Output of the whole image> A case where the entire image is output will be described. An example of the entire image is shown in FIG. 5. As shown in FIG. 5, the entire image 130 includes unitary images corresponding to each unitary optical system 31 (images obtained by photoelectrically converting light from the subject that is incident via each unitary optical system 31). For example, the image in the center of the entire image 130 is a unitary image corresponding to the unitary optical system 310. The image in the upper right corner of the entire image 130 is a unitary image corresponding to the unitary optical system 311. The image in the upper left corner of the entire image 130 is a unitary image corresponding to the unitary optical system 312. The image in the lower left corner of the entire image 130 is a unitary image corresponding to the unitary optical system 313. The image in the lower right corner of the entire image 130 is a unitary image corresponding to the unitary optical system 314.

[0118] The entire image 130 may be the entire captured image generated by the image sensor 51, or a partial image (including all unitary images) cut out from the captured image. The entire image 130 may be an image in RAW format or an image in YC format.

[0119] Based on the viewpoint area information, a portion (effective portion) of each unitary image is designated as a viewpoint image area for the entire image 130. For example, in the case of FIG. 5, the area of ​​the entire image 130 surrounded by a dotted line frame is the viewpoint image area. That is, a portion (effective portion) of the unitary image corresponding to the unitary optical system 310 is designated as the viewpoint image area 1310. Similarly, a portion (effective portion) of the unitary image corresponding to the unitary optical system 311 is designated as the viewpoint image area 1311. Furthermore, a portion (effective portion) of the unitary image corresponding to the unitary optical system 312 is designated as the viewpoint image area 1312. Furthermore, a portion (effective portion) of the unitary image corresponding to the unitary optical system 313 is designated as the viewpoint image area 1313. Furthermore, a portion (effective portion) of the unitary image corresponding to the unitary optical system 314 is designated as the viewpoint image area 1314. In the following, when it is not necessary to distinguish between the viewpoint image areas 1310 to 1314, they will be referred to as the viewpoint image area 131.

[0120] When outputting such an overall image 130, the associating unit 70 acquires the overall image 130 from the image sensor 51, the RAW signal processing unit 52, or the camera signal processing unit 54, and associates the viewpoint-related information corresponding to the multi-eye optical system 30, which is supplied from the region identifying unit 56, with the overall image 130. The associating unit 70 then outputs the overall image and viewpoint-related information associated with each other. As an example of output, for example, the storage unit 62 may store the overall image and viewpoint-related information associated with each other in the storage medium 63. Alternatively, the communication unit 64 may transmit the overall image and viewpoint-related information associated with each other. Furthermore, the filing unit 65 may file the overall image and viewpoint-related information associated with each other.

[0121] The association of the overall image with the viewpoint-related information may be performed in the area extraction unit 53. That is, the area extraction unit 53 may associate the viewpoint-related information supplied from the area identification unit 56 with the overall image to be output, and supply the overall image and viewpoint-related information associated with each other to the bus 60, the RAW signal processing unit 52, or the camera signal processing unit 54.

[0122] In this case, the viewpoint-related information includes viewpoint area information indicating a plurality of viewpoint image areas in the captured image. The viewpoint area information may represent the viewpoint image area in any way. For example, the viewpoint image area may be represented by coordinates indicating a position corresponding to the optical axis of the single optical system 31 in the captured image (center coordinates of the viewpoint image area) and the resolution (number of pixels) of the viewpoint image (viewpoint image area). In other words, the viewpoint area information may include the center coordinates of the viewpoint image area in the captured image and the resolution of the viewpoint image area. In this case, the location of the viewpoint image area in the entire image 130 can be identified from the center coordinates of the viewpoint image area and the resolution (number of pixels) of the viewpoint image area.

[0123] By associating such viewpoint-related information with a captured image, this viewpoint-related information can be used in extracting viewpoint images as pre-processing for subsequent processing such as depth estimation by multi-eye matching and processing for suppressing errors that occur when attaching (installing) the multi-eye optical system 30. For example, the image reconstruction processing unit 57 can extract each viewpoint image based on viewpoint region information included in this viewpoint-related information, and then perform subsequent processing such as depth estimation by multi-eye matching, refocusing processing, and processing for suppressing errors that occur when attaching (installing) the multi-eye optical system 30.

[0124] Note that even if viewpoint-related information is not associated with the overall image 130, for example, the image reconstruction processing unit 57 may be able to identify a viewpoint image area included in the overall image 130 by image processing, but it may be difficult to accurately identify the viewpoint image area within the captured image depending on the imaging conditions, etc. Therefore, by associating viewpoint-related information with the overall image 130 as described above, the image reconstruction processing unit 57 can more easily and more accurately extract the viewpoint image area from the above-mentioned overall image 130 based on the viewpoint-related information.

[0125] <Output of viewpoint images> Next, a case where viewpoint images are output will be described. Fig. 6 is a diagram showing an example of a cut-out viewpoint image. In Fig. 6, viewpoint image 1320 is an image obtained by extracting viewpoint image region 1310 from overall image 130 (Fig. 5). Viewpoint image 1321 is an image obtained by extracting viewpoint image region 1311 from overall image 130. Viewpoint image 1322 is an image obtained by extracting viewpoint image region 1312 from overall image 130. Viewpoint image 1323 is an image obtained by extracting viewpoint image region 1313 from overall image 130. Viewpoint image 1324 is an image obtained by extracting viewpoint image region 1314 from overall image 130. In the following, when it is not necessary to distinguish between viewpoint images 1320 to 1324, they will be referred to as viewpoint image 132.

[0126] When outputting such viewpoint images, the area extraction unit 53 outputs each viewpoint image 132 cut out as in the example of FIG. 6 as independent data (or files).

[0127] For example, the region extraction unit 53 cuts out viewpoint images from the captured image (whole image) in accordance with viewpoint-related information supplied from the region identification unit 56. The region extraction unit 53 assigns viewpoint identification information (e.g., an identification number) to each of the cut-out viewpoint images to identify each viewpoint. The region extraction unit 53 supplies each viewpoint image to which the viewpoint identification information has been assigned to the camera signal processing unit 54. The camera signal processing unit 54 performs camera signal processing on each viewpoint image in the RAW format to generate each viewpoint image in the YC format. The camera signal processing unit 54 supplies each viewpoint image in the YC format to the associating unit 70. Furthermore, the region identification unit 56 supplies the viewpoint-related information supplied to the region extraction unit 53 to the associating unit 70.

[0128] The associating unit 70 associates, with each viewpoint image, viewpoint-related information corresponding to that viewpoint image. The viewpoint-related information may include viewpoint identification information (e.g., a viewpoint identification number) for identifying each viewpoint. The associating unit 70 associates, with each viewpoint image, viewpoint-related information corresponding to that viewpoint image based on this viewpoint identification information. By referring to this viewpoint identification information, the associating unit 70 can easily determine which viewpoint-related information corresponds to which viewpoint image. In other words, by using this viewpoint identification information, the associating unit 70 can more easily correctly associate each viewpoint image with the viewpoint-related information.

[0129] The associating unit 70 then outputs the associated viewpoint images and viewpoint-related information. For example, the storage unit 62 may store the associated viewpoint images and viewpoint-related information in a storage medium 63. The communication unit 64 may transmit the associated viewpoint images and viewpoint-related information. Furthermore, the filing unit 65 may file the associated viewpoint images and viewpoint-related information.

[0130] The association of each viewpoint image with the viewpoint-related information may be performed in the area extraction unit 53. That is, the area extraction unit 53 may associate the viewpoint-related information supplied from the area identification unit 56 with each viewpoint image to be output, and supply the associated viewpoint images and viewpoint-related information to the bus 60, the RAW signal processing unit 52, or the camera signal processing unit 54.

[0131] Furthermore, the viewpoint-related information may include viewpoint time information indicating the time and order in which the captured image from which the viewpoint image is extracted was captured. When viewpoint images extracted from a plurality of captured images are mixed, or when the captured images are moving images or continuous images, it may be difficult to identify which viewpoint image was extracted from which captured image. By associating viewpoint time information indicating the generation time and order of the captured images with the viewpoint images, it is possible to more easily identify the captured image corresponding to each viewpoint image (the captured image from which each viewpoint image was extracted). In other words, it is possible to more easily identify multiple viewpoint images extracted from the same captured image. In addition, even when recorded files are not managed collectively, it is possible to identify each viewpoint image at the same time.

[0132] As in the case of viewpoint images, unit images may be cut out from the captured image and processed or recorded.

[0133] <Output of composite image> Next, a case where a composite image is output will be described. Fig. 7 is a diagram showing an example of a composite image obtained by combining each viewpoint image. In the example of Fig. 7, one composite image 133 is generated by combining viewpoint images 1320 to 1324 extracted in the example of Fig. 6 so that they are displayed side by side in one image. In other words, the composite image 133 is obtained by combining each viewpoint image 132 and converting them into one data (one frame) or one file.

[0134] 7, blank areas are shown around the viewpoint images 1320 to 1324 of the composite image 133, but the composite image 133 may or may not have these blank areas. Furthermore, the shape of the composite image 133 only needs to be rectangular, and the arrangement (alignment) of the viewpoint images 132 is arbitrary. As in the example of FIG. 7, a blank area (an area corresponding to the sixth viewpoint image 132) that occurs when five viewpoint images 132 are aligned in two rows and three columns may be represented by null data or a fixed value.

[0135] For example, the area extraction unit 53 cuts out viewpoint images from the captured image (whole image) in accordance with viewpoint-related information supplied from the area identification unit 56. The area extraction unit 53 generates a composite image by combining the cut-out viewpoint images so that they are displayed side by side within a single image. At this time, by determining the order (position) of the viewpoint images in advance, it is possible to easily determine which viewpoint each viewpoint image included in the composite image is from.

[0136] Furthermore, viewpoint identification information (e.g., an identification number) may be assigned to each viewpoint image before the images are synthesized. In this case, it is also possible to easily determine which viewpoint each viewpoint image included in the synthesized image is from. In the following, it is assumed that the order of the viewpoint images in the synthesized image is predetermined.

[0137] The region extraction unit 53 supplies the composite image to which the viewpoint identification information has been assigned to the camera signal processing unit 54. The camera signal processing unit 54 performs camera signal processing on the composite image in RAW format to generate a composite image in YC format. The camera signal processing unit 54 supplies the composite image in YC format to the associating unit 70. In addition, the region identification unit 56 supplies the viewpoint-related information supplied to the region extraction unit 53 to the associating unit 70.

[0138] The associating unit 70 associates viewpoint-related information with the composite image. The viewpoint of each viewpoint image included in the composite image is clear from the position of that viewpoint image in the composite image. In other words, it is easy to understand which viewpoint area information in the viewpoint-related information each viewpoint image corresponds to.

[0139] The associating unit 70 then outputs the associated composite image and viewpoint-related information. For example, the storage unit 62 may store the associated composite image and viewpoint-related information in the storage medium 63. The communication unit 64 may transmit the associated composite image and viewpoint-related information. Furthermore, the filing unit 65 may file the associated image and viewpoint-related information.

[0140] The composite image and the viewpoint-related information may be associated with each other in the area extraction unit 53. That is, the area extraction unit 53 may associate the viewpoint-related information supplied from the area identification unit 56 with the composite image to be output, and supply the associated composite image and viewpoint-related information to the bus 60, the RAW signal processing unit 52, or the camera signal processing unit 54.

[0141] <Capturing the subject> For example, as in the example of Fig. 8, it is assumed that camera 10 is used to capture images of subject 141 and subject 142. As shown in Fig. 8, it is assumed that subject 141 is located closer (in front) than subject 142 when viewed from camera 10.

[0142] 9 is an example of an overall image obtained as a captured image or a photographed image by the camera 10. As in the example of FIG. 5, the overall image 130 includes unitary images corresponding to the individual unitary optical systems 31 (i.e., each unitary optical system 31 is used as a viewpoint).

[0143] For example, the subjects 1410 and 1420 in the overall image 130 are images of the subjects 141 and 142 generated by the image sensor 51 performing photoelectric conversion on light received via the unitary optical system 310. That is, the subjects 1410 and 1420 are images of the subjects 141 and 142 (corresponding to the unitary optical system 310) taken from the unitary optical system 310 as the viewpoint. The subjects 1411 and 1421 in the overall image 130 are images of the subjects 141 and 142 generated by the image sensor 51 performing photoelectric conversion on light received via the unitary optical system 311. That is, the subjects 1411 and 1421 are images of the subjects 141 and 142 (corresponding to the unitary optical system 311) taken from the unitary optical system 311 as the viewpoint.

[0144] Similarly, subjects 1412 and 1422 in overall image 130 are images of subjects 141 and 142 generated by image sensor 51 performing photoelectric conversion on light received via unitary optical system 312. That is, subjects 1412 and 1422 are images of subjects 141 and 142 (corresponding to unitary optical system 312) taken from unitary optical system 312 as a viewpoint. Furthermore, subjects 1413 and 1423 in overall image 130 are images of subjects 141 and 142 generated by image sensor 51 performing photoelectric conversion on light received via unitary optical system 313. That is, subjects 1413 and 1423 are images of subjects 141 and 142 (corresponding to unitary optical system 313) taken from unitary optical system 313 as a viewpoint. Furthermore, the subjects 1414 and 1424 in the overall image 130 are images of the subjects 141 and 142 generated by the image sensor 51 performing photoelectric conversion on the light received via the unitary optical system 314. In other words, the subjects 1414 and 1424 are images of the subjects 141 and 142 (corresponding to the unitary optical system 314) with the unitary optical system 314 as the viewpoint.

[0145] <Image display> The camera 10 can display the image, for example, on the display panel unit 33 or the viewfinder unit 34. For example, the camera 10 can generate a display image from the captured image and display the display image as a through image so that the user can check the composition before capturing an image of the subject by, for example, pressing the shutter button. The camera 10 performs such a display for each frame of the captured image captured by the image sensor 51. The user can frame (adjust the angle of view) while checking the image displayed in this way (also referred to as a display image), thereby more easily capturing an image according to the user's intentions.

[0146] Also, for example, camera 10 can generate a display image from the captured image and display the display image as a confirmation image so that the user can check the image capture result (i.e., the captured image) immediately after capturing an image of a subject by pressing the shutter button, etc. Furthermore, for example, camera 10 can generate a display image from an image saved in storage medium 83 or the like and display the display image so that the user can check captured images that were captured and saved in the past.

[0147] Generally, when capturing an image using a monocular optical system, the entire image formed on the image sensor 51 is displayed as a through image. However, in the case of an entire image 130 ( FIG. 9 ) captured using a plurality of unitary optical systems 31, the entire image 130 includes unitary images having each unitary optical system 31 as a viewpoint. Therefore, when the entire image 130 is displayed on the display unit 61 (display panel unit 33, viewfinder unit 34, etc.) as a through image, each unitary image becomes much smaller than when capturing an image using a monocular optical system, which may make it difficult for the user to visually recognize the subject in the displayed image. Therefore, for example, it may be difficult for the user to adjust (framing) the angle of view (composition) based on the displayed image.

[0148] The same applies when displaying a confirmation image of a captured image or a display image of a saved image, and in the case of the entire image 130, it may be difficult for the user to visually recognize the subject in the displayed image.

[0149] Therefore, a method of cutting out and displaying one ommatidium image has been devised. For example, in the case of Fig. 9, an area including subjects 1410 and 1420 is cut out, and an image of that area is displayed on the display unit 61 (display panel unit 33, viewfinder unit 34, etc.) as a through image or the like. In this way, the subject can be displayed enlarged on the display unit 61, allowing the user to visually recognize the subject more easily.

[0150] However, with this method, since only one ommatidium image is displayed, it is difficult for the user to confirm the degree of parallax of each ommatidium image based on the displayed image.

[0151] <Rotating display of multiple ommatidial images> Therefore, the ommatidium images to be displayed are dynamically switched. For example, in the case of the whole image 130 in Fig. 9, an ommatidium cutout image 1510, an ommatidium cutout image 1511, an ommatidium cutout image 1512, an ommatidium cutout image 1513, and an ommatidium cutout image 1514 are cut out.

[0152] Unitary cutout image 1510 is a part or all of the unitary image corresponding to unitary optical system 310, and includes subject 1410 and subject 1420. Unitary cutout image 1511 is a part or all of the unitary image corresponding to unitary optical system 311, and includes subject 1411 and subject 1421. Unitary cutout image 1512 is a part or all of the unitary image corresponding to unitary optical system 312, and includes subject 1412 and subject 1422. Unitary cutout image 1513 is a part or all of the unitary image corresponding to unitary optical system 313, and includes subject 1413 and subject 1423. Unitary cutout image 1514 is a part or all of the unitary image corresponding to unitary optical system 314, and includes subject 1414 and subject 1424.

[0153] In the following, when it is not necessary to distinguish between the ommatidium cutout image 1514 and the ommatidium cutout image 1514, they will be referred to as the ommatidium cutout image 151. Furthermore, the region of the ommatidium cutout image 151 in the image before cutting (for example, the entire image 130) will also be referred to as the ommatidium cutout region.

[0154] Then, some or all of these ommatidium cutout images 151 are dynamically switched and displayed in a predetermined order. For example, as shown in Fig. 10, the ommatidium cutout images 151 are repeatedly displayed in the order of ommatidium cutout image 1510, ommatidium cutout image 1513, ommatidium cutout image 1512, ommatidium cutout image 1511, and ommatidium cutout image 1514.

[0155] Since each unitary cutout image 151 has a different viewpoint position, parallax occurs between the images. Therefore, as shown in Fig. 11, when unitary cutout images 1510 to 1514 are superimposed on one another at the same position, the positions of subjects 1420 to 1424 are shifted from one another. In other words, when each unitary cutout image 151 is displayed consecutively, the user sees that subject 142 is blurred, as shown in Fig. 11. The user can intuitively grasp the degree of parallax with which the subject is captured based on the magnitude of this blur.

[0156] In this case, each individual image 151 can be displayed larger than when the entire image 130 is displayed, thereby preventing a decrease in visibility. Therefore, adjustment (framing) of the angle of view (composition) can be more easily performed based on this displayed image. In other words, it is possible to realize ease of viewing the subject and framing, as well as to easily grasp the parallax of the images from each individual image.

[0157] As described above, one of a plurality of images having a plurality of unit optical systems, each having an independent optical path, as a viewpoint is selectively and dynamically switched and displayed on the display unit.

[0158] For example, an information processing device may be provided with a display control unit that selectively and dynamically switches between multiple images viewed from multiple single-eye optical systems whose optical paths are independent of each other, and displays one of these images on a display unit.

[0159] Furthermore, for example, a program can be used to cause a computer to function as a display control unit that selectively and dynamically switches between and displays on a display unit one of a plurality of images whose viewpoints are multiple single-eye optical systems whose optical paths are independent of each other.

[0160] By doing so, it is possible to more easily grasp the parallax between a plurality of images taken from different unit optical systems as viewpoints.

[0161] For example, as in the example of FIG. 10, some or all of the multiple images may be selected and displayed one by one in a predetermined order.

[0162] For example, when displaying a through image, the camera 10 (display image generating unit 55) extracts and displays the individual ommatidial cutout images 151 one by one from each frame of the captured image, which is a moving image.

[0163] Furthermore, if the captured image is a still image, when displaying a confirmation image of the captured image, the camera 10 (display image generation unit 55) extracts a plurality of unitary cutout images 151 from the captured image, which is a still image, and displays them sequentially. The same applies when displaying a saved captured image. Note that, when the captured image is a moving image, the camera 10 (display image generation unit 55) extracts and displays one unitary cutout image 151 from each frame, as in the case of a through image.

[0164] <Ommatidia segmentation area> Next, the unitary cutout regions cut out as described above will be described. In the entire image 130, a part or all of the region of each unitary image is set as the unitary cutout region. In other words, the number and rough positions of the unitary cutout regions set in the entire image 130 are the same as those of the unitary images.

[0165] The size of the unitary cutout region is arbitrary as long as it does not exceed the unitary image. For example, the size of the unitary cutout region may be the same as the size of the viewpoint image region. The size of the unitary cutout region may also be variable (for example, it may be set by the user), but will be described here as being fixed. Furthermore, for ease of viewing the displayed image, it is desirable that the sizes of the unitary cutout regions are the same.

[0166] The shape of the ommatidium cutout regions is arbitrary. Here, the description will be made assuming that they are rectangular. Furthermore, in order to make the displayed image easier to view, it is desirable that the shapes of the ommatidium cutout regions are the same.

[0167] <Shifting the ommatidial segmentation area> In the example of Fig. 9, the position of the subject 141 is the same (located at the center) in each unitary cutout image 151. Therefore, when these unitary cutout images 151 are displayed consecutively, the subject 141 does not blur, as in the example of Fig. 10. In other words, by making the position of the desired subject the same in each unitary cutout image 151, the subject can be displayed without blur.

[0168] As described above, each unitary image has parallax, and the position of the subject in each unitary image shifts in the direction of the parallax. In other words, by shifting the position of each unitary cutout region in the unitary image in the direction of the parallax and aligning the positions of the desired subject in each unitary cutout image 151, the subject can be displayed without blurring.

[0169] The direction of this shift (direction of parallax) depends on the arrangement pattern of the unitary cut-out images 151 (or unitary images), that is, the layout (number, positions, etc.) of the unitary optical systems 31 in the multi-eye optical system 30.

[0170] Here, the parallax of unitary images will be explained. Generally, the closer an object is to the camera 10, the larger the parallax between unitary images, and the farther an object is from the camera 10, the smaller the parallax. For example, in the case of Figure 9, the positional deviation of object 141 in each unitary image is larger than the positional deviation of object 142.

[0171] Therefore, the larger the shift amount of the individual cutout areas, the more the blur width of a subject closer to the camera 10 can be reduced, and the smaller the shift amount of the individual cutout areas, the more the blur width of a subject farther from the camera 10 can be reduced. In other words, the shift amount of the individual cutout areas depends on the distance between the camera 10 and the subject for which the blur width is to be reduced (typically, for which the subject is to be displayed without blur).

[0172] This shift amount may be controlled by a user, an application, etc. As described above, the shift amount is determined by the distance from camera 10 to the subject whose blur width is to be reduced on the display screen, so by a user or an application specifying this shift amount, it is possible to reduce the blur width of a subject that is at a distance from camera 10 that corresponds to that shift amount.

[0173] For example, as shown in Fig. 12, the shift amount may be specified as vector information (direction and magnitude). In the example of Fig. 12, the direction and magnitude of the shift of the area of ​​unitary cutout image 1511 are specified by shift amount 1611, which is vector information. Similarly, the direction and magnitude of the shift of the area of ​​unitary cutout image 1512 are specified by shift amount 1612, the direction and magnitude of the shift of the area of ​​unitary cutout image 1513 are specified by shift amount 1613, and the direction and magnitude of the shift of the area of ​​unitary cutout image 1514 are specified by shift amount 1614. Hereinafter, when it is not necessary to distinguish between the shift amounts 1611 to 1614, they will be referred to as shift amount 161. Note that the area of ​​unitary cutout image 1510 is located at the center of the entire image 130 and is therefore used as a reference and is not shifted (is fixed).

[0174] For example, a user or an application sets a shift amount 161 corresponding to the distance between a desired subject and the camera 10. For example, when a user or an application specifies a distance, the shift amount 161 corresponding to the specified distance may be set. The individual cutout regions are shifted in accordance with this setting, and the blur width of the desired subject in the displayed image is suppressed (typically, blur can be eliminated). In other words, the camera 10 can reduce the blur width of a subject at an arbitrary distance from the camera 10 in the displayed image.

[0175] Fig. 13 shows an example in which the unitary cutout regions are shifted so as to reduce the blur width of the subject 142. In the example of Fig. 13, the position of the subject 1420 in the unitary cutout image 1510, the position of the subject 1421 in the unitary cutout image 1511, the position of the subject 1422 in the unitary cutout image 1512, the position of the subject 1423 in the unitary cutout image 1513, and the position of the subject 1424 in the unitary cutout image 1514 are all the same. In other words, the positions of the subject 142 in each unitary cutout image 151 are the same.

[0176] Therefore, when these unitary cutout images 151 are displayed sequentially, the displayed images will look like Fig. 14. That is, as shown in Fig. 14, camera 10 can display unitary cutout images 151 so that subject 142 appears to the user without blurring and subject 141 appears to be blurred.

[0177] The method for measuring the distance between the camera 10 and the subject is arbitrary. For example, the user may visually measure and specify the distance between the camera 10 and the subject, or the distance between the camera 10 and the subject may be measured using a distance measurement sensor or the like.

[0178] The shift amount may be fixed. For example, the shift amount may be fixed to a value corresponding to a recommended shooting distance according to the optical characteristics of the single optical system 31 (lens). In this case, the photographer can adjust the distance between the camera 10 and the subject so that the parallax of the subject (i.e., the movement of the subject in the displayed image) is reduced, thereby enabling the photographer to capture an image at a distance that matches the recommended shooting distance. In other words, the user can use this display function to easily measure the distance to the subject.

[0179] <Switching cycle> When the ommatidium cutout images 151 to be displayed are dynamically switched as described above, the ommatidium cutout images 151 to be displayed may be switched at predetermined intervals.

[0180] The switching period can be any period, for example, it can be every frame (single frame) or every multiple frames.

[0181] The switching cycle may also be changeable by the user (photographer). For example, the user may set the switching cycle by operating an input device such as the dial 35 or button 36 of the camera 10. For example, the control unit 81 may accept the switching cycle specification input by the user in this manner, and supply control information indicating the switching cycle to the display image generation unit 55.

[0182] <Display order of ommatidium cutout images> An example of the selection order (display order) of the unitary cutout images 151 when dynamically switching and displaying such unitary cutout images 151 is shown in Fig. 10, but this selection order is arbitrary and is not limited to the example of Fig. 10. As in the example of Fig. 10, the camera 10 may be configured to sequentially display all unitary cutout images 151, or as in the example of Fig. 15, the camera 10 may be configured to display some of the unitary cutout images 151 in a predetermined order.

[0183] 15, the display image generation unit 55 does not extract or display the unitary cut-out image 1510. The display image generation unit 55 extracts the unitary cut-out images 151 from the captured image (or captured image) in the order of unitary cut-out image 1513, unitary cut-out image 1512, unitary cut-out image 1511, and unitary cut-out image 1514, and displays them on the display unit 61.

[0184] In other words, in this case, the display image generation unit 55 extracts (part or all of) the multiple ommatidium cut-out images 151 that are located on the outer periphery in terms of their relative positional relationship one by one in a predetermined order, and displays them on the display unit 61.

[0185] By doing so, in the display image displayed by the display unit 61, the subject 142 appears to move in a rectangular shape. Therefore, when the user follows the movement of the subject 142, the direction of movement of the user's line of sight is limited to the up-down and left-right directions, and the orientation is also limited. In other words, the movement of the user's line of sight is simplified to a movement that describes a rectangular shape. Therefore, the user can follow the movement of the subject 142 more easily than if the position of the subject 142 changes randomly. In other words, the subject 142 appears smoother.

[0186] Furthermore, in the example of Fig. 15, unitary cutout image 151 of the outer periphery is displayed, and this display can represent the entire range in which the position of subject 142, which appears blurred, changes. For example, displaying only unitary cutout image 1510 and unitary cutout image 1513 can only represent a part of the range in which the position of subject 142 changes. For example, the position of subject 142 in unitary cutout image 1511 cannot be represented. This may make it difficult to grasp the actual extent of parallax. In contrast, by displaying as in the example of Fig. 15, the user can more accurately grasp the magnitude of parallax.

[0187] Furthermore, one unitary cutout image 151 may be displayed multiple times during one display cycle of the unitary cutout images 151 as shown in Figures 10 and 15. For example, the unitary cutout images 151 may be extracted and displayed in the order shown in Figures 16 and 17. In the examples of Figures 16 and 17, unitary cutout images 1510 are displayed twice during one display cycle of the unitary cutout images 151.

[0188] Furthermore, some or all of the multiple unitary cutout images 151 may be selected and displayed one by one in an order in which the scanning trajectory is line-symmetrical in the relative positional relationship of the multiple unitary cutout images 151 (at each viewpoint). For example, as shown in Fig. 16, the display image generation unit 55 may extract the unitary cutout images 151 from the captured image (or captured image) in the order of unitary cutout image 1510, unitary cutout image 1513, unitary cutout image 1512, unitary cutout image 1510, unitary cutout image 1511, and unitary cutout image 1514, and display them on the display unit 61.

[0189] By doing so, the movement of the subject 142 appears line-symmetrical in the left-right and up-down directions in the displayed image. Therefore, when the user follows the movement of the subject 142, the movement of the user's line of sight also becomes line-symmetrical, improving the continuity of the viewpoint movement. Therefore, the user can follow the movement of the subject 142 more easily than if the position of the subject 142 changes randomly. In other words, the subject 142 appears smoother.

[0190] Furthermore, some or all of the multiple unitary cutout images 151 may be selected and displayed one by one in an order in which the scanning trajectory is rotationally symmetric in the relative positional relationship of the multiple unitary cutout images 151 (at each viewpoint). For example, as shown in Fig. 17, the display image generation unit 55 may extract the unitary cutout images 151 from the captured image (or captured image) in the order of unitary cutout image 1510, unitary cutout image 1513, unitary cutout image 1512, unitary cutout image 1510, unitary cutout image 1514, and unitary cutout image 1511, and display them on the display unit 61.

[0191] By doing so, the movement of subject 142 appears rotationally symmetric (point symmetric in this case) in the displayed image. Therefore, when a user follows the movement of subject 142, the movement of the user's line of sight also becomes rotationally symmetric (point symmetric), improving the continuity of the viewpoint movement. Therefore, the user can follow the movement of subject 142 more easily than if the position of subject 142 changes randomly. In other words, subject 142 appears smoother.

[0192] In the examples of Figures 16 and 17, all unitary cut-out images 151 are displayed, so the user can check, for example, whether there is an abnormality in any one of the unitary optical systems 31 (for example, dust adhering to the lens).

[0193] Furthermore, the selection order (display order) of the unitary cutout images 151 may be other than the above-described example. For example, the start position of one cycle (the first unitary cutout image 151) may be different from the above-described example. That is, in each of the examples in Fig. 10 and Fig. 15 to Fig. 17, one cycle may start from a display sequence number other than "0."

[0194] The patterns of the selection order (display order) in each of the above examples may also be rotated. For example, the display image generation unit 55 may rotate the pattern in the example of Fig. 10 and extract the unitary cutout images 151 from the captured image (or captured image) in the order of unitary cutout image 1510, unitary cutout image 1511, unitary cutout image 1514, unitary cutout image 1513, and unitary cutout image 1512, and display them on the display unit 61. Furthermore, the display image generation unit 55 may rotate the pattern in the example of Fig. 16 and extract the unitary cutout images 151 from the captured image (or captured image) in the order of unitary cutout image 1510, unitary cutout image 1512, unitary cutout image 1511, unitary cutout image 1510, unitary cutout image 1514, and unitary cutout image 1513, and display them on the display unit 61. In this case, too, the movement of subject 142 appears to be line-symmetric in the left-right and up-down directions in the displayed image. Furthermore, for example, display image generating unit 55 may rotate the pattern in the example of Fig. 17 and extract unitary cutout images 151 from the captured image (or captured image) in the order of unitary cutout image 1510, unitary cutout image 1512, unitary cutout image 1511, unitary cutout image 1510, unitary cutout image 1513, and unitary cutout image 1514, and display these unitary cutout images on display unit 61. In this case, the movement of subject 142 appears to be rotationally symmetric (point symmetric) in the displayed image.

[0195] Furthermore, the patterns of the selection order (display order) in each of the above examples may be inverted upside down. For example, the display image generation unit 55 may invert the pattern in the example of Fig. 10 upside down, extract the unitary cutout images 151 from the captured image (or captured image) in the order of unitary cutout image 1510, unitary cutout image 1512, unitary cutout image 1513, unitary cutout image 1514, and unitary cutout image 1511, and display them on the display unit 61. Also, for example, the display image generation unit 55 may invert the pattern in the example of Fig. 15 upside down, extract the unitary cutout images 151 from the captured image (or captured image) in the order of unitary cutout image 1512, unitary cutout image 1513, unitary cutout image 1514, and unitary cutout image 1511, and display them on the display unit 61. 16 upside down, extracting the unitary cutout images 151 from the captured image (or the captured image) in the order of unitary cutout image 1510, unitary cutout image 1512, unitary cutout image 1513, unitary cutout image 1510, unitary cutout image 1514, and unitary cutout image 1511, and displaying them on the display unit 61. Also, the display image generating unit 55 may upside down the pattern in the example of FIG. 17 upside down, extracting the unitary cutout images 151 from the captured image (or the captured image) in the order of unitary cutout image 1510, unitary cutout image 1512, unitary cutout image 1513, unitary cutout image 1510, unitary cutout image 1511, and unitary cutout image 1514, and displaying them on the display unit 61.

[0196] The patterns of the selection order (display order) in each of the above examples may be flipped horizontally. For example, the display image generation unit 55 may flip the pattern in the example of Fig. 10 horizontally, extract the unitary cutout images 151 from the captured image (or captured image) in the order of unitary cutout image 1510, unitary cutout image 1514, unitary cutout image 1511, unitary cutout image 1512, and unitary cutout image 1513, and display them on the display unit 61. Also, the display image generation unit 55 may flip the pattern in the example of Fig. 15 horizontally, extract the unitary cutout images 151 from the captured image (or captured image) in the order of unitary cutout image 1514, unitary cutout image 1511, unitary cutout image 1512, and unitary cutout image 1513, and display them on the display unit 61. 16 horizontally, extracting the unitary cutout images 151 from the captured image (or the captured image) in the order of unitary cutout image 1510, unitary cutout image 1514, unitary cutout image 1511, unitary cutout image 1510, unitary cutout image 1512, and unitary cutout image 1513, and displaying them on the display unit 61. Also, the display image generating unit 55 may horizontally reverse the pattern of the example of FIG. 17 horizontally, extracting the unitary cutout images 151 from the captured image (or the captured image) in the order of unitary cutout image 1510, unitary cutout image 1514, unitary cutout image 1511, unitary cutout image 1510, unitary cutout image 1513, and unitary cutout image 1512, and displaying them on the display unit 61.

[0197] Of course, the direction of inversion is arbitrary, and may be a direction other than the above-mentioned up-down and left-right directions (that is, a diagonal direction).

[0198] Furthermore, the images may be selected (displayed) in the reverse order of the selection order (display order) of each example described above. For example, the display image generation unit 55 may reverse the order of the patterns in the example of Fig. 10 , extract the unitary cutout images 151 from the captured image (or captured image) in the order of unitary cutout image 1514, unitary cutout image 1511, unitary cutout image 1512, unitary cutout image 1513, and unitary cutout image 1510, and display them on the display unit 61. Also, for example, the display image generation unit 55 may reverse the order of the patterns in the example of Fig. 15 , extract the unitary cutout images 151 from the captured image (or captured image) in the order of unitary cutout image 1514, unitary cutout image 1511, unitary cutout image 1512, and unitary cutout image 1513, and display them on the display unit 61. 16 , extracting the unitary cutout images 151 from the captured image (or captured image) in the order of unitary cutout image 1514, unitary cutout image 1511, unitary cutout image 1510, unitary cutout image 1512, unitary cutout image 1513, and unitary cutout image 1510, and displaying them on the display unit 61. Also, the display image generating unit 55 may reverse the order of the pattern in the example of FIG. 17 , extracting the unitary cutout images 151 from the captured image (or captured image) in the order of unitary cutout image 1511, unitary cutout image 1514, unitary cutout image 1510, unitary cutout image 1512, unitary cutout image 1513, and unitary cutout image 1510, and displaying them on the display unit 61.

[0199] Furthermore, the above-mentioned methods of rotating or inverting the pattern, reversing the order, etc. may be combined as appropriate.

[0200] The selection order (display order) of the unitary cutout images 151 as described above may be specified by a user or an application. For example, the user may operate the dial 35 or the button 36 to specify the selection order (display order) of the unitary cutout images 151. Also, for example, an application may specify the selection order (display order) of the unitary cutout images 151 according to the operation mode of the camera 10, etc. For example, multiple candidates for the selection order (display order) of the unitary cutout images 151 may be prepared in advance, and the user or the application may specify the order to be applied from among them. Also, the user or the application may be able to set one cycle in any order.

[0201] <Display image generation unit> 18 is a block diagram showing an example of the main configuration of the display image generation unit 55 (FIG. 3). As shown in FIG. 18, the display image generation unit 55 has a display ommatidium selection unit 201, a shift amount determination unit 202, an ommatidium cut-out region setting unit 203, and a cut-out processing unit 204.

[0202] The display unit selection unit 201 performs processing related to the selection of unit images. That is, the display unit selection unit 201 performs processing related to the selection of the unit optical system 31. That is, the display unit selection unit 201 performs processing related to the selection of the viewpoint.

[0203] For example, the display ommatidium selection unit 201 can acquire viewpoint-related information (VI or VI') supplied from the area identification unit 56. This viewpoint-related information includes information indicating the area (coordinates) of each ommatidium image and identification information of each ommatidium image. This identification information may be any information, and may be, for example, an identification number (also referred to as an ommatidium number) for identifying an ommatidium image. In the following description, it is assumed that the ommatidium number is included in the viewpoint-related information as this identification information.

[0204] Based on this viewpoint-related information, the display unit selection unit 201 can grasp the number of unit images included in the entire image, as well as the area and unit number of each unit image. Therefore, based on the viewpoint-related information, the display unit selection unit 201 can select unit images from which unit cut-out images 151 to be displayed as images for display are extracted. That is, based on this viewpoint-related information, the display unit selection unit 201 can select the unit optical system 31 from which the image for display is obtained. That is, based on this viewpoint-related information, the display unit selection unit 201 can select the viewpoint of the image for display.

[0205] For example, the display unitary selection unit 201 selects unitary images (unitary optical systems 31, or viewpoints) in a predetermined order or in an order designated by a user or an application based on this viewpoint-related information. For example, the display unitary selection unit 201 may acquire control information indicating designation by a user, an application, or the like from the control unit 81, and may select unitary images based on the control information. For example, the display unitary selection unit 201 may acquire control information related to a switching pattern (selection order) or switching cycle of unitary images from the control unit 81, and may select unitary images based on the control information.

[0206] The display ommatidium selection unit 201 supplies the ommatidium number indicating the selected ommatidium image to the shift amount determination unit 202 and the ommatidium cutout region setting unit 203 .

[0207] That is, the display unit selection unit 201 selectively and dynamically switches and displays one of a plurality of unit cut-out images, the viewpoints of which are a plurality of unit optical systems whose optical paths are independent of each other, on the display unit. That is, the display unit selection unit 201 performs display control of the image for display.

[0208] The shift amount determination unit 202 performs processing related to determining the shift amount (direction and size) of the unitary cutout region. For example, the shift amount determination unit 202 can acquire the unitary number supplied from the display unitary selection unit 201. This unitary number is identification information that specifies the unitary image to be displayed (i.e., the unitary image from which the unitary cutout image 151 is extracted). In addition, the shift amount determination unit 202 can acquire viewpoint-related information (VI or VI') supplied from the region identification unit 56. This viewpoint-related information includes information indicating the region (coordinates) of each unitary image and the unitary number of each unitary image.

[0209] Furthermore, the shift amount determination unit 202 can acquire shift amount control information supplied from the control unit 81. This shift amount control information includes information used to determine the shift amount. For example, the shift amount control information may include information indicating a distance (the distance between the camera 10 and the subject) specified by a user, an application, or the like. The shift amount control information may also include information regarding an operation mode specified by a user, an application, or the like, such as whether the shift amount is variable or fixed. For example, when the control unit 81 receives a specification regarding the shift amount from a user or an application, the control unit 81 generates shift amount control information including that information and supplies the information to the shift amount determination unit 202.

[0210] The shift amount determination unit 202 determines the arrangement of each ommatidium image based on the supplied viewpoint-related information, and determines which ommatidium image has been selected based on the supplied ommatidium number. This allows the shift amount determination unit 202 to determine the direction of shift of the ommatidium cut-out region. Furthermore, the shift amount determination unit 202 can determine the magnitude of shift of the ommatidium cut-out region based on the shift amount control information. For example, the shift amount determination unit 202 can determine the magnitude of the shift amount to be a magnitude according to the distance specified in the shift amount control information.

[0211] The shift amount determination unit 202 can determine the direction and magnitude of the shift of the ommatidium cut-out region as described above and generate the shift amount (vector information). The shift amount determination unit 202 can supply the shift amount to the ommatidium cut-out region setting unit 203.

[0212] That is, the shift amount determination unit 202 can control the shift amount (direction and size) of the position of the ommatidium cut-out region.

[0213] The unitary cut-out region setting unit 203 performs processing related to setting of unitary cut-out regions. For example, the unitary cut-out region setting unit 203 can acquire viewpoint-related information (VI or VI') supplied from the region specifying unit 56. This viewpoint-related information includes information indicating the region (coordinates) of each unitary image and the unitary number of each unitary image.

[0214] Furthermore, the ommatidium cutout region setting unit 203 can acquire the ommatidium number supplied from the display ommatidium selection unit 201. This ommatidium number is identification information that specifies the ommatidium image to be displayed (i.e., the ommatidium image from which the ommatidium cutout image 151 is extracted).

[0215] Furthermore, the unitary cut-out region setting unit 203 can acquire the shift amount (vector information) supplied from the shift amount determination unit 202. This shift amount is information indicating the direction and magnitude of the shift of the unitary cut-out region.

[0216] The unitary cutout region setting unit 203 can identify the region of the unitary image corresponding to the supplied unitary number (e.g., the center coordinates and range (resolution) of the region) based on the viewpoint-related information. Then, the unitary cutout region setting unit 203 can set the initial value of the image cutout region (e.g., the center coordinates and range (resolution) of the region) corresponding to the unitary image. Furthermore, the unitary cutout region setting unit 203 can shift the image cutout region based on the shift amount and update the position of the image cutout region.

[0217] The ommatidium cutout region setting unit 203 can supply the cutout processing unit 204 with cutout coordinates, which are coordinate information indicating the image cutout region set in this way.

[0218] The cutout processing unit 204 performs processing related to extraction of unitary cutout images. For example, the cutout processing unit 204 can acquire the entire image of the captured image supplied from the camera signal processing unit 54. The cutout processing unit 204 can also acquire cutout coordinates supplied from the unitary cutout region setting unit 203.

[0219] The cutout processing unit 204 can extract an area specified by the cutout coordinates from the acquired captured image (whole image). The cutout processing unit 204 can supply the extracted ommatidium cutout image to the display unit 61 as a display image (through image).

[0220] That is, the cutout processing unit 204 cuts out a unitary cutout image from a unitary image having the same unitary optical system as the unitary image selected by the display unitary selection unit 201 as its viewpoint.

[0221] The cutout processing unit 204 can also extract unitary cutout images from viewpoint images or composite images. In this case, the cutout processing unit 204 acquires viewpoint images or composite images.

[0222] The display image generation unit 55 can also generate a confirmation image of the captured image or a display image of a saved captured image. In this case, the cropping processing unit 204 can acquire the captured image from the camera signal processing unit 54 or the storage unit 62 and extract individual cropped images from the captured image. The captured image may include an entire image, a viewpoint image, and a composite image generated from the captured image.

[0223] With the above configuration, the display image generating unit 55 can dynamically switch the unit images to be displayed, which allows the user to more easily understand the parallax between multiple images taken from different unit optical systems as viewpoints by checking the displayed images.

[0224] The cutout processing unit 204 may cut out unitary cutout regions of an image in RAW format to generate unitary cutout images in RAW format. In this case, color separation processing and YC conversion may be performed on the unitary cutout images in RAW format to generate unitary cutout images in YC format, and the unitary cutout images in YC format may be supplied to the display unit 61. For example, the unitary cutout images in RAW format cut out by the cutout processing unit 204 may be returned to the camera signal processing unit 54, where color separation processing and YC conversion may be performed.

[0225] <Image capture process flow> Next, a process flow relating to image display by such camera 10 will be described. Note that the following description will be given taking the display of a through image as an example. For example, when a user turns on the power of camera 10 or switches the operation mode of camera 10 to an imaging mode for capturing images, imaging processing begins. Figure 19 is a flowchart illustrating an example of the flow of this imaging processing.

[0226] When the imaging process is started, the display ommatidium selection unit 201, shift amount determination unit 202, and ommatidium cutout region setting unit 203 of the display image generation unit 55 acquire and set viewpoint-related information supplied from the region identification unit 56 in step S101.

[0227] In step S102, the shift amount determination unit 202 acquires and sets the shift amount control information supplied from the control unit 81.

[0228] In step S103, the display image generating unit 55 and the display unit 61 execute a through image display process to generate and display a through image. Details of this through image display process will be described later.

[0229] In step S104, the control unit 81 determines whether to perform preparation for imaging. For example, when a user, an application, or the like performs control related to imaging, such as control of focus, aperture, imaging mode, etc., the control unit 81 determines to perform preparation processing for imaging (also referred to as imaging preparation processing) in accordance with that control. In that case, the processing proceeds to step S105.

[0230] In step S105, the control unit 81 performs imaging preparation processing by appropriately controlling each processing unit, such as the optical system control unit 84. This imaging preparation processing may be any processing related to imaging. For example, it may include at least one of focus and aperture adjustment, imaging mode setting, flash setting, image quality setting, etc., or may include other processing.

[0231] When the process of step S105 ends, the process proceeds to step S106. Also, if it is determined in step S104 that the imaging preparation process is not to be performed, the process proceeds to step S106.

[0232] In step S106, the control unit 81 determines whether or not to capture an image. For example, if it is determined that an image is to be captured, such as when the user presses the shutter button, the process proceeds to step S107.

[0233] In step S107, the image sensor 51, the area extraction unit 53, the camera signal processing unit 54, etc. capture an image of the subject to generate a captured image, and generate an entire image, a viewpoint image, or a composite image from the captured image.

[0234] In step S108, the storage unit 62 and the communication unit 64 output the entire image, viewpoint image, or composite image generated in step S107. For example, the storage unit 62 stores the image (entire image, viewpoint image, composite image) in the storage medium 63. The communication unit 64 supplies the image (entire image, viewpoint image, composite image) to a device external to the camera 10. Note that the image may be output after being filed by the filing unit 65.

[0235] When the process of step S108 ends, the process proceeds to step S109. If it is determined in step S106 that imaging is not to be performed, the process proceeds to step S109.

[0236] In step S109, the control unit 81 determines whether or not to end the imaging process. If it is determined that the imaging process should not be ended, the process returns to step S103, and the subsequent processes are repeated.

[0237] That is, the processes in steps S103 to S109 are executed for each frame. If it is determined in step S109 that the imaging process should be ended, the imaging process ends.

[0238] <Through image display process flow> Next, an example of the flow of the through image display process executed in step S103 of Fig. 19 will be described with reference to the flowchart of Fig. 20. Note that here, a case will be described in which individual cut-out images are extracted from the entire captured image.

[0239] When the through image display process is started, the display unit selection unit 201 executes the display unit selection process in step S121, and selects the unit images to be displayed based on the viewpoint-related information and the like.

[0240] In step S122, the shift amount determination unit 202 determines the shift amount (direction and size) of the ommatidium cut-out region to be extracted from the ommatidium image selected in step S121 based on the viewpoint-related information, the shift amount control information, and the like.

[0241] In step S123, the unitary cut-out region setting unit 203 sets unitary cut-out regions (cut-out coordinates) to be extracted from the unitary image selected in step S121 based on the viewpoint-related information, the shift amount set in step S122, and the like.

[0242] In step S 124 , the cutout processing unit 204 acquires the entire image of the captured image generated by the area extraction unit 53 .

[0243] In step S125, the cutout processing unit 204 cuts out the ommatidium cutout region set in step S123 from the entire image acquired in step S124, and generates an ommatidium cutout image.

[0244] In step S126, the display unit 61 displays the ommatidium cut-out images extracted as described above as through images.

[0245] When the process of step S126 ends, the through image display process ends, and the process returns to FIG.

[0246] <Flow of display ommatidium selection process> Next, an example of the flow of the display ommatidium selection process executed in step S121 of FIG. 20 will be described with reference to the flowchart of FIG.

[0247] When the display unit selection process starts, the display unit selection unit 201 determines in step S141 whether the number of displayed frames has reached the unit switching period. Here, the number of displayed frames is a variable indicating how many frames of the same unit cut-out image have been displayed. The unit switching period is a setting value that specifies how many frames of the same unit cut-out image are to be displayed. This unit switching period may be a predetermined fixed value, or may be set by the user, an application, or the like.

[0248] If it is determined that the number of frames that have already been displayed has reached the unitary switching cycle, that is, if it is determined that it is time (frame) to switch the unitary cut-out images to be displayed, the process proceeds to step S142.

[0249] In step S142, the display ommatidium selection unit 201 sets the number of displayed frames (variable) to an initial value (for example, "1"). Note that this initial value is arbitrary and may be other than "1", such as "0".

[0250] In step S143, the display unit selection unit 201 updates the display sequence number. The display sequence number is a number (variable) indicating the display order in the switching pattern of the unit cut-out images, and is updated as shown in the following formula (1).

[0251] Display sequence number = (Display sequence number + 1) mod Display sequence number ···(1)

[0252] Here, the number of display sequences is the number of unitary cutout images that make up one cycle of the switching pattern of unitary cutout images, and the display sequence number is a number that indicates the order of the current processing target within that one cycle of the sequence.

[0253] For example, in the case of FIG. 10, one cycle of the switching pattern of the unitary cutout images 151 is composed of five unitary cutout images 151. Therefore, the number of display sequences in this case is "5", and the display sequence number is any one of "0" to "4" (i.e., any one of five values). In the example of FIG. 15, one cycle of the switching pattern of the unitary cutout images 151 is composed of four unitary cutout images 151. Therefore, the number of display sequences in this case is "4", and the display sequence number is any one of "0" to "3" (i.e., any one of four values). In the examples of FIGS. 16 and 17, one cycle of the switching pattern of the unitary cutout images 151 is composed of six unitary cutout images 151. Therefore, the number of display sequences in this case is "6", and the display sequence number is any one of "0" to "5" (i.e., any one of six values).

[0254] In equation (1), the function mod represents the remainder of the division. That is, the value of the display sequence number is incremented by 1 by the processing of step S143, and when it reaches the display sequence number, it is reset to "0." That is, in the case of FIG. 10, for example, the display sequence number repeats the values ​​"0" to "4" as follows: "0" → "1" → "2" → "3" → "4" → "0" → ...

[0255] In step S144, the display unit selection unit 201 acquires the unit number corresponding to the display sequence number updated by the processing in step S143 from the table information. The display unit selection unit 201 has table information indicating one cycle of the switching pattern of the unit cut-out images. This table information indicates which unit images (unit cut-out images) are to be displayed and in what order, and indicates the correspondence between each display sequence number and the unit number. Based on the table information, the display unit selection unit 201 identifies the unit number corresponding to the updated display sequence number.

[0256] If the switching pattern of the unitary cut-out images is variable, that is, if the pattern to be applied can be set by the user or an application, the display unitary selection unit 201 may specify the unitary number based on table information corresponding to the applied pattern. For example, the display unitary selection unit 201 may have multiple types of table information candidates in advance, and select table information corresponding to an instruction from the control unit 81 (pattern specification by the user or an application) from the candidates.

[0257] Furthermore, the display ommatidium selection unit 201 may generate table information corresponding to the applied pattern based on information from the control unit 81. For example, the display ommatidium selection unit 201 may generate table information based on criteria such as the ommatidium with the maximum vertical coordinate, the ommatidium with the maximum horizontal coordinate, the ommatidium with the minimum vertical coordinate, and the ommatidium with the minimum horizontal coordinate.

[0258] In step S145, the display ommatidium selection unit 201 sets the ommatidium number as the display ommatidium number. The display ommatidium number indicates the ommatidium image to be displayed. That is, the display ommatidium selection unit 201 supplies this display ommatidium number to the shift amount determination unit 202 and the ommatidium cutout region setting unit 203.

[0259] When the process of step S145 ends, the display ommatidium selection process ends, and the process returns to FIG.

[0260] Also, if it is determined in step S141 that the number of displayed frames has not reached the unitary switching cycle, that is, if it is determined that it is not the timing (frame) to switch the unitary cut-out images to be displayed, the process proceeds to step S146.

[0261] In step S146, the display ommatidium selection unit 201 increments the number of frames already displayed (variable) by "+1" without switching (updating) the ommatidium cut-out image.

[0262] In step S147, the display ommatidium selection unit 201 sets the display ommatidium number from the previous frame (the previous frame) as the display ommatidium number for this frame. In other words, the display ommatidium number is not updated. As a result, the display ommatidium selection unit 201 supplies the same display ommatidium number as that from the previous frame to the shift amount determination unit 202 and the ommatidium cutout region setting unit 203.

[0263] When the process of step S147 ends, the display ommatidium selection process ends, and the process returns to FIG.

[0264] By performing each process as described above, the display image generating unit 55 can dynamically switch the unit images to be displayed, which allows the user to more easily understand the parallax between multiple images taken from different unit optical systems as viewpoints by checking the displayed images.

[0265] When displaying a confirmation image of a captured image or a saved captured image, the captured image may be subjected to the same processing as the through image display processing described above.

[0266] <Number of ommatidia> The number of unitary optical systems 31 constituting the multi-eye optical system 30 (number of units) may be any number as long as it is plural, and is not limited to the above-mentioned five units (five unitary optical systems 31). The number of units may be either an odd number or an even number. The arrangement pattern (relative positional relationship) of each unitary optical system 31 is also arbitrary.

[0267] <7 eyes> For example, the number of units may be seven (seven unit optical systems 31). In this case, an example of the entire image of the captured image (or captured image) generated by the camera 10 is shown in FIG.

[0268] 22, the whole image 130 in this case includes seven subjects 141 and seven subjects 142. That is, seven unitary images are included in the whole image 130. That is, in the case of the whole image 130, seven types of unitary cut-out images 151 can be extracted, such as unitary cut-out images 1510 to 1516 shown in FIG.

[0269] In this case, the switching pattern of the unitary cutout images is also arbitrary, as in the case of five lenses. For example, a switching pattern in which all the unitary optical systems 31 are selected once each may be used. For example, one cycle may be a pattern in which the images are switched in the order of unitary cutout image 1510 → unitary cutout image 1511 → unitary cutout image 1512 → unitary cutout image 1513 → unitary cutout image 1514 → unitary cutout image 1515 → unitary cutout image 1516 (→ unitary cutout image 1510).

[0270] Also, for example, a switching pattern may be used to select (a part or all of) the unitary cutout image 151 located on the outer periphery in terms of their relative positional relationship among these unitary cutout images 151. For example, one cycle may be a pattern of switching in the order of unitary cutout image 1511 → unitary cutout image 1512 → unitary cutout image 1513 → unitary cutout image 1514 → unitary cutout image 1515 → unitary cutout image 1516 (→ unitary cutout image 1511).

[0271] Furthermore, for example, a switching pattern may be used in which some or all of these unitary cutout images 151 are selected one by one in an order in which the relative positional relationship between them results in a scanning trajectory that is line-symmetrical in any direction. For example, the images may be line-symmetrical in the vertical direction, the horizontal direction, or the diagonal direction. For example, one cycle may be a pattern in which the images are switched in the following order: unitary cutout image 1510 → unitary cutout image 1511 → unitary cutout image 1512 → unitary cutout image 1513 → unitary cutout image 1510 → unitary cutout image 1514 → unitary cutout image 1515 → unitary cutout image 1516 → (→ unitary cutout image 1510).

[0272] Furthermore, for example, a switching pattern may be used in which some or all of these unitary cutout images 151 are selected one by one in an order in which the scanning trajectory is rotationally symmetric (for example, point symmetric) in their relative positional relationship. For example, one cycle may be a pattern in which switching is performed in the order of unitary cutout image 1510 → unitary cutout image 1511 → unitary cutout image 1512 → unitary cutout image 1513 → unitary cutout image 1510 → unitary cutout image 1516 → unitary cutout image 1515 → unitary cutout image 1514 → (→ unitary cutout image 1510). Furthermore, one cycle may be a pattern of switching in the following order: ommatidium cutout image 1510 → ommatidium cutout image 1511 → ommatidium cutout image 1512 → ommatidium cutout image 1510 → ommatidium cutout image 1513 → ommatidium cutout image 1514 → ommatidium cutout image 1510 → ommatidium cutout image 1515 → ommatidium cutout image 1516 (→ ommatidium cutout image 1510).

[0273] Furthermore, for example, a minimum switching pattern may be used that selects some or all of these unitary cutout images 151 so that the maximum parallax in the up-down direction and the maximum parallax in the left-right direction can be grasped. For example, one cycle may be a pattern that switches in the order of unitary cutout image 1511 → unitary cutout image 1512 → unitary cutout image 1514 → unitary cutout image 1515 (→ unitary cutout image 1511). In other words, from the multiple unitary cutout images, multiple images in which the parallax between the unitary cutout images is larger than the parallax between the other images may be selected, and the selected multiple images may be selected and displayed one by one.

[0274] As in the case of five eyes, the patterns of the selection order (display order) of each of the above examples may be rotated. Also, the patterns of the selection order (display order) of each of the above examples may be inverted in any direction (up and down, left and right, or diagonally). Furthermore, the patterns may be selected (displayed) in the reverse order of the selection order (display order) of each of the above examples. Also, the above methods of rotating, inverting, reversing the order, etc. may be combined as appropriate. The patterns of each of the above examples may be specified by the user or an application.

[0275] <9 eyes> For example, the number of units may be nine (nine unit optical systems 31). In this case, an example of the entire image of the captured image (or captured image) generated by the camera 10 is shown in FIG.

[0276] 23, the whole image 130 in this case includes nine subjects 141 and nine subjects 142. That is, nine unitary images are included in the whole image 130. That is, in the case of the whole image 130, nine types of unitary cut-out images 151 can be extracted, such as unitary cut-out images 1510 to 1518 shown in FIG.

[0277] In this case, the switching pattern of the unitary cutout images is also arbitrary, as in the case of five lenses. For example, a switching pattern in which all the unitary optical systems 31 are selected once each may be used. For example, one cycle may be a pattern in which the images are switched in the following order: unitary cutout image 1510 → unitary cutout image 1511 → unitary cutout image 1512 → unitary cutout image 1513 → unitary cutout image 1514 → unitary cutout image 1515 → unitary cutout image 1516 → unitary cutout image 1517 → unitary cutout image 1518 (→ unitary cutout image 1510).

[0278] Furthermore, for example, a switching pattern may be used to select (a part or all of) the unitary cutout image 151 located on the outer periphery in terms of their relative positional relationship among these unitary cutout images 151. For example, one cycle may be a pattern of switching in the order of unitary cutout image 1511 → unitary cutout image 1512 → unitary cutout image 1513 → unitary cutout image 1514 → unitary cutout image 1515 → unitary cutout image 1516 → unitary cutout image 1517 → unitary cutout image 1518 (→ unitary cutout image 1511).

[0279] Furthermore, for example, a switching pattern may be used in which some or all of these unitary cutout images 151 are selected one by one in an order in which the relative positional relationship between them results in a line symmetry of the scanning trajectory in any direction. For example, the images may be line symmetric in the vertical direction, the horizontal direction, or the diagonal direction. For example, one cycle may be a pattern in which the images are switched in the following order: unitary cutout image 1510 → unitary cutout image 1518 → unitary cutout image 1511 → unitary cutout image 1512 → unitary cutout image 1513 → unitary cutout image 1514 → unitary cutout image 1510 → unitary cutout image 1514 → unitary cutout image 1515 → unitary cutout image 1516 → unitary cutout image 1517 → unitary cutout image 1518 (→ unitary cutout image 1510).

[0280] Alternatively, for example, a switching pattern may be used in which some or all of these unitary cutout images 151 are selected one by one in an order in which the scanning trajectory is rotationally symmetric (for example, point symmetric) in their relative positional relationship. For example, one cycle may be a pattern in which the images are switched in the order of unitary cutout image 1510 → unitary cutout image 1518 → unitary cutout image 1511 → unitary cutout image 1512 → unitary cutout image 1513 → unitary cutout image 1514 → unitary cutout image 1510 → unitary cutout image 1518 → unitary cutout image 1517 → unitary cutout image 1516 → unitary cutout image 1515 → unitary cutout image 1514 (→ unitary cutout image 1510). Furthermore, one cycle may be a pattern of switching in the following order: ommatidium cutout image 1510 → ommatidium cutout image 1511 → ommatidium cutout image 1512 → ommatidium cutout image 1510 → ommatidium cutout image 1513 → ommatidium cutout image 1514 → ommatidium cutout image 1510 → ommatidium cutout image 1515 → ommatidium cutout image 1516 → ommatidium cutout image 1510 → ommatidium cutout image 1517 → ommatidium cutout image 1518 (→ ommatidium cutout image 1510).

[0281] Furthermore, for example, a switching pattern may be used to select some or all of these unitary cutout images 151 so that the maximum parallax in the up-down direction and the maximum parallax in the left-right direction can be grasped. For example, one cycle may be a pattern of switching in the order of unitary cutout image 1511 → unitary cutout image 1513 → unitary cutout image 1515 → unitary cutout image 1517 (→ unitary cutout image 1511).

[0282] Furthermore, for example, some or all of these unitary cutout images 151 may be switched in a horizontal figure-eight pattern that allows selection so that the maximum parallax in the up-down direction and the maximum parallax in the left-right direction can be grasped. For example, one cycle may be a pattern in which the unitary cutout images 1511 → unitary cutout image 1513 → unitary cutout image 1517 → unitary cutout image 1515 (→ unitary cutout image 1511) are switched in this order.

[0283] Furthermore, for example, a minimum configuration switching pattern may be selected so that the maximum parallax in the up-down direction and the maximum parallax in the left-right direction can be grasped for some or all of these ommatidium cutout images 151. For example, one cycle may be a pattern in which the ommatidium cutout image 1511 is switched to the ommatidium cutout image 1515 (→ the ommatidium cutout image 1511) in that order.

[0284] As in the case of five eyes, the patterns of the selection order (display order) of each of the above examples may be rotated. Also, the patterns of the selection order (display order) of each of the above examples may be inverted in any direction (up and down, left and right, or diagonally). Furthermore, the patterns may be selected (displayed) in the reverse order of the selection order (display order) of each of the above examples. Also, the above methods of rotating, inverting, reversing the order, etc. may be combined as appropriate. The patterns of each of the above examples may be specified by the user or an application.

[0285] <2. Second embodiment> <Camera system> In the first embodiment, the present technology has been described using the camera 10 equipped with the multi-eye optical system 30 as an example, but the present technology can also be applied to other configurations. For example, an optical system including the multi-eye optical system 30 may be replaceable. In other words, the multi-eye optical system 30 may be configured to be detachable from the camera 10.

[0286] <Camera system appearance> Fig. 24 is a perspective view showing an example configuration of an embodiment of a camera system to which the present technology is applied. The camera system 301 shown in Fig. 24 is composed of a camera body 310 and a multi-eye interchangeable lens 320 (lens unit). When the multi-eye interchangeable lens 320 is attached to the camera body 310, the camera system 301 has the same configuration as the camera 10 and basically performs the same processing. In other words, the camera system 301 functions as an imaging device that captures an image of a subject and generates image data of the captured image, similar to the camera 10.

[0287] The camera body 310 is designed so that the multi-eye interchangeable lens 320 can be attached and detached. That is, the camera body 310 has a camera mount 311, and the multi-eye interchangeable lens 320 (the lens mount 322) is attached to the camera mount 311, thereby mounting the multi-eye interchangeable lens 320 on the camera body 310. Note that the camera body 310 may also be designed so that general interchangeable lenses other than the multi-eye interchangeable lens 320 can be attached and detached.

[0288] The camera body 310 has a built-in image sensor 51. The image sensor 51 receives light rays focused by the multi-lens interchangeable lens 320 or other interchangeable lenses attached to (the camera mount 311 of) the camera body 310 and performs photoelectric conversion to capture an image of a subject.

[0289] The multi-eye interchangeable lens 320 has a lens barrel 321 and a lens mount 322. The multi-eye interchangeable lens 320 also has a plurality of five unit optical systems 310, 311, 312, 313, and 314.

[0290] As in the case of the camera 10, the multiple unitary optical systems 31 in this case are configured so that the optical paths of light passing through them are independent of one another. That is, light that has passed through each unitary optical system 31 irradiates different positions on the light-receiving surface (e.g., effective pixel area) of the image sensor 51 without entering the other unitary optical systems 31. At least the optical axes of the unitary optical systems 31 are located at different positions on the light-receiving surface of the image sensor 51, and at least a portion of the light that has passed through each unitary optical system 31 irradiates different positions on the light-receiving surface of the image sensor 51.

[0291] Therefore, similarly to the case of the camera 10, in the captured image generated by the image sensor 51 (the entire image output by the image sensor 51), images of the subject formed via each unit optical system 31 are formed at different positions. In other words, from the captured image, captured images (also referred to as viewpoint images) with each unit optical system 31 as a viewpoint are obtained. That is, by attaching the multi-eye interchangeable lens 320 to the camera body 310 and capturing an image of the subject, a plurality of viewpoint images can be obtained.

[0292] The lens barrel 321 has a substantially cylindrical shape, and is formed on one bottom side of the cylindrical shape with a lens mount 322. The lens mount 322 is attached to the camera mount 311 of the camera body 310 when the multi-eye interchangeable lens 320 is attached to the camera body 310.

[0293] The five unit optical systems 31 are provided in the multi-eye interchangeable lens 320 in a two-dimensional plane orthogonal to the lens barrel optical axis (parallel to the light receiving surface (imaging surface) of the image sensor 51), with the unit optical system 310 at the center (center of gravity) and the other four unit optical systems 311 to 314 arranged to form the vertices of a rectangle. Of course, the arrangement shown in Figure 24 is just one example, and the positional relationship of each unit optical system 31 is arbitrary as long as the optical paths are independent of each other.

[0294] <Example of electrical configuration of camera system> FIG. 25 is a block diagram showing an example of the electrical configuration of the camera system 301 of FIG.

[0295] <Camera body> In the camera system 301, the camera body 310 has an image sensor 51, a RAW signal processing unit 52, an area extraction unit 53, a camera signal processing unit 54, a display image generation unit 55, an area identification unit 56, an image reconstruction processing unit 57, a bus 60, a display unit 61, a storage unit 62, a communication unit 64, a filing unit 65, a control unit 81, and a storage unit 82. In other words, the camera body 310 has a configuration other than the multi-eye optical system 30 and the optical system control unit 84 provided in the lens barrel 20 of the camera 10.

[0296] In addition to the above-described configuration, the camera body 310 also has a communication unit 341. This communication unit 341 is a processing unit that communicates with (the communication unit 351 of) the multi-eye interchangeable lens 320 when it is properly attached to the camera body 310, and transmits and receives information. The communication unit 341 can communicate with the multi-eye interchangeable lens 320 using any communication method. This communication may be wired communication or wireless communication.

[0297] For example, the communication unit 341 is controlled by the control unit 81, performs this communication, and acquires information supplied from the multi-eye interchangeable lens 320. Also, for example, the communication unit 341 supplies information supplied from the control unit 81 to the multi-eye interchangeable lens 320 through this communication. Any information can be exchanged with the multi-eye interchangeable lens 320. For example, it may be data, or it may be control information such as commands or control parameters.

[0298] <Multi-lens interchangeable lens> In the camera system 301, the multi-eye interchangeable lens 320 has a communication unit 351 and a storage unit 352 in addition to the multi-eye optical system 30 and the optical system control unit 84. The communication unit 351 communicates with the communication unit 341 when the multi-eye interchangeable lens 320 is properly attached to the camera body 310. This communication enables the exchange of information between the camera body 310 and the multi-eye interchangeable lens 320. The communication method of the communication unit 351 is arbitrary, and may be wired communication or wireless communication. Furthermore, the information exchanged through this communication may be data, or may be control information such as commands and control parameters.

[0299] For example, the communication unit 351 acquires control information transmitted from the camera body 310 via the communication unit 341. The communication unit 351 supplies the acquired information to the optical system control unit 84 as necessary, so that the information can be used to control the multi-eye optical system 30.

[0300] Furthermore, the communication unit 351 can supply the acquired information to the storage unit 352, which can store the information in the storage medium 353. The communication unit 351 can also read out the information stored in the storage medium 353 via the storage unit 352, and transmit the information to the camera body 310 (communication unit 341).

[0301] The storage medium 353 may be a ROM or a rewritable memory such as a RAM or flash memory. In the case of a rewritable memory, the storage medium 353 can store any information.

[0302] <Memory of viewpoint-related information 1> In the camera system 301 configured as described above, the location where the viewpoint-related information corresponding to the multi-eye interchangeable lens 320 (i.e., the multi-eye optical system 30) is stored is arbitrary. For example, it may be stored in a storage medium 353 of the multi-eye interchangeable lens 320. Then, for example, the control unit 81 of the camera body 310 may access the storage unit 352 via the communication units 351 and 341 and read the viewpoint-related information from the storage medium 353. Then, the viewpoint-related information may be set in the area identification unit 56 via the control unit 81, and further set in the display image generation unit 55.

[0303] For example, such processing may be performed at any timing or trigger prior to imaging, such as when the multi-lens interchangeable lens 320 is properly attached to the camera body 310, when the power is turned on to the camera system 301, or when the driving mode of the camera system 301 transitions to an imaging mode in which an image of a subject can be captured.

[0304] In this way, the camera body 310 can perform image processing using viewpoint images by using viewpoint-related information corresponding to the multi-eye interchangeable lens 320 (i.e., the multi-eye optical system 30). In other words, the display image generation unit 55 can correctly select unit images and appropriately set shift amounts of unit cut-out regions corresponding to the multi-eye interchangeable lens 320 attached to the camera body 310. In other words, in the case of this camera system 301, as in the case of the camera 10, the display image generation unit 55 can dynamically switch the unit images to be displayed. This allows the camera 500 to perform displays similar to those in the case of the camera 10. Therefore, by checking the displayed images, the user can more easily grasp the parallax between multiple images taken from different unit optical systems as viewpoints.

[0305] <Memory of viewpoint-related information 2> Furthermore, the control unit 81 may supply the viewpoint-related information of the multi-eye interchangeable lens 320 acquired from that multi-eye interchangeable lens 320 to the storage unit 82 together with identification information (hereinafter referred to as ID) of that multi-eye interchangeable lens 320, for storage. In this case, the storage unit 82 associates the supplied identification information with the viewpoint-related information and stores them in the storage medium 83. In other words, the camera body 310 can manage the viewpoint-related information and ID of the multi-eye interchangeable lens 320. Therefore, the camera body 310 can manage the viewpoint-related information of a plurality of multi-eye interchangeable lenses 320.

[0306] By doing this, from the next time onwards, the control unit 81 can read out the viewpoint-related information corresponding to the ID from the storage unit 82 (storage medium 83) by acquiring the ID of the multi-eye interchangeable lens 320. In other words, the control unit 81 can easily acquire the viewpoint-related information corresponding to the multi-eye interchangeable lens 320.

[0307] <Memory of viewpoint-related information 3> Furthermore, the storage medium 83 may store in advance the viewpoint-related information of a plurality of multi-eye interchangeable lenses 320 in association with the IDs of the multi-eye interchangeable lenses 320. In other words, in this case, the camera body 310 manages the viewpoint-related information of a plurality of multi-eye interchangeable lenses 320 in advance.

[0308] By doing this, the control unit 81 can easily read out the viewpoint-related information corresponding to the ID from the memory unit 82 (storage medium 83) using the ID of the multi-eye interchangeable lens 320 that is correctly attached to the camera body 310.

[0309] <3. Third Embodiment> <Multiple image sensors> In the above description, one image sensor 51 receives light that has passed through a plurality of unit optical systems 31 and performs photoelectric conversion to generate a captured image, but this is not limiting, and different image sensors may receive light that has passed through each unit optical system 31. An example of the configuration in this case is shown in FIG.

[0310] 26 , camera 500 is one embodiment of an imaging device to which the present technology is applied. Camera 500 is basically the same device as camera 10, has the same configuration as camera 10, and performs the same processing. However, camera 500 has unitary imaging units 5110 to 5114 instead of the multi-eye optical system 30 and image sensor 51 of camera 10. Unitary imaging unit 5110 has unitary optical system 310 and image sensor 510. Unitary imaging unit 5111 has unitary optical system 311 and image sensor 511. Unitary imaging unit 5112 has unitary optical system 312 and image sensor 512. Unitary imaging unit 5113 has unitary optical system 313 and image sensor 513. Unitary imaging unit 5114 has unitary optical system 314 and image sensor 514. In the following description, the unitary imaging units 5110 to 5114 will be referred to as unitary imaging unit 511 when there is no need to distinguish between them.

[0311] That is, the camera 500 has a plurality of unitary imaging units 511. Light that has passed through the unitary optical system 31 of each unitary imaging unit 511 is incident on the image sensor 51 of that unitary imaging unit 511 and is photoelectrically converted to generate a captured image. That is, a captured image of a unitary image is generated in each unitary imaging unit 511. The captured image (unitary image) generated in the image sensor 51 of each unitary imaging unit 511 is supplied to the RAW signal processing unit 52, area extraction unit 53, area identification unit 56, bus 60, etc., as in the case of FIG. 3.

[0312] 3, the optical system control unit 84 controls the lens groups and apertures of the unit optical systems 31 of each unit imaging unit 511. The control unit 81 also controls each unit imaging unit 511 (the image sensor 51) to capture an image of the subject.

[0313] In such a configuration, the camera signal processing unit 54 supplies the unitary images to the display image generation unit 55. The display image generation unit 55 generates a display image from the unitary images. In this case, the display image generation unit 55 also dynamically switches the unitary images to be used as display images. This allows the camera 500 to perform a display similar to that of the camera 10. Therefore, by checking the display image, the user can more easily grasp the parallax between multiple images taken from different unitary optical systems.

[0314] <Display image generation unit> Fig. 27 is a block diagram showing an example of the main configuration of the display image generation unit 55 in this case. As shown in Fig. 27, in this case, the display image generation unit 55 has an image selection unit 521 in addition to the configuration of the camera 10 shown in Fig. 18.

[0315] The image selection unit 521 acquires and holds each unit image (captured image generated by each unit image capturing unit 511) supplied from the camera signal processing unit 54. Then, when the image selection unit 521 acquires an unit number (an unit number indicating an unit image selected by the display unit image capturing unit 201) supplied from the display unit selection unit 201, the image selection unit 521 supplies the unit image corresponding to the unit number to the cutout processing unit 204.

[0316] In this case, the display ommatidium selection unit 201 supplies the ommatidium number of the selected ommatidium image to the shift amount determination unit 202, the ommatidium cutout region setting unit 203, and the image selection unit 521.

[0317] When the cropping processing unit 204 acquires the ommatidium image supplied from the image selection unit 521, it extracts from the ommatidium image an ommatidium cropping region indicated by the cropping coordinates supplied from the ommatidium cropping region setting unit 203. In this case, the cropping processing unit 204 converts the coordinates in the ommatidium image or the coordinates in the entire image into either one to appropriately extract the image. The cropping processing unit 204 supplies the cropped ommatidium cropped image to the display unit 61 for display.

[0318] <Through image display process flow> An example of the flow of the through image display process in this case will be described with reference to the flowchart in Fig. 28. When the through image display process starts, the processes of steps S501 to S503 are executed in the same manner as the processes of steps S121 to S123 in Fig. 20.

[0319] In step S504, the image selection unit 521 selects an ommatidium image corresponding to the ommatidium number (display ommatidium number) indicating the ommatidium image selected in the display ommatidium selection process (FIG. 21) in step S501.

[0320] In step S505, the cutout processing unit 204 extracts the ommatidium cutout region set in step S503 from the ommatidium image selected in step S504, and generates a ommatidium cutout image.

[0321] The process of step S506 is executed in the same manner as the process of step S126 (FIG. 20). When the process of step S506 ends, the process returns to FIG.

[0322] In this way, the display image generation unit 55 can dynamically switch the unit images to be used as display images, thereby enabling the display unit 61 to perform the same display as in the case of the camera 10. Therefore, by checking the displayed image, the user can more easily grasp the parallax between multiple images taken from different unit optical systems.

[0323] Of course, in this case, the present technology can also be applied to the display of a confirmation image of a captured image or a saved captured image, similar to the display of a through image described above. Note that, when displaying a confirmation image of a captured image or a saved captured image, it is sufficient to perform the same processing as the through image display processing described above on the captured image.

[0324] As in the case of the multi-eye optical system 30, the number of unitary imaging units 511 (number of units) is arbitrary, and may be either an odd number or an even number.

[0325] <Drive control of the individual image capture unit> Furthermore, in the case of the camera 500, the drive of each unitary imaging unit 511 can be controlled individually. Therefore, for example, when displaying a through image, unitary images may be selected by controlling (selecting) the unitary imaging unit 511 to be driven. In other words, the drive of the unitary imaging unit 511 that generates unitary images not selected as images to be displayed may be stopped (i.e., the unitary images may not be generated). At this time, the control unit 81 may stop controlling the aperture and focus of the unitary imaging unit 511 whose drive is stopped. Furthermore, the control unit 81 may stop controlling the white balance gain of the image obtained by the unitary imaging unit 511 whose drive is stopped. In this way, the number of unitary imaging units 511 to be driven can be reduced, and an increase in power consumption can be suppressed.

[0326] <Display image generation unit> Fig. 29 is a block diagram showing an example of the main configuration of the display image generation unit 55 in this case. As shown in Fig. 29, in this case, the display image generation unit 55 has the same configuration as in the case of the camera 10 shown in Fig. 18.

[0327] In this case, however, the display unit selection unit 201 supplies the unit number of the selected unit image as power control information to the control unit 81. The control unit 81 controls the optical system control unit 84 based on the power control information to stop driving the unit image capturing unit 511 corresponding to the unit image not selected by the display unit selection unit 201.

[0328] The optical system control unit 84, in accordance with the control of the control unit 81, stops driving the unitary imaging units 511 corresponding to the unitary images not selected by the display unitary selection unit 201. As a result, only the unitary imaging units 511 that generate the unitary images selected by the display unitary selection unit 201 are driven, and only the unitary images selected by the display unitary selection unit 201 are generated. In other words, the unitary images selected by the display unitary selection unit 201 are supplied to the display image generation unit 55.

[0329] The cutout processing unit 204 extracts an ommatidium cutout region from the supplied ommatidium image (the ommatidium image selected by the display ommatidium selection unit 201), generates an ommatidium cutout image, and supplies it to the display unit 61 to be displayed.

[0330] <Through image display process flow> An example of the flow of the through image display process in this case will be described with reference to the flowchart in Fig. 30. When the through image display process starts, the processes of steps S521 to S523 are executed in the same manner as the processes of steps S501 to S503 in Fig. 28.

[0331] In step S524, the display unit selection unit 201 drives the unit image capturing unit 511 corresponding to the unit number (display unit number) indicating the unit image selected by the display unit selection process (FIG. 21) of step S501. The cutout processing unit 204 acquires the unit image selected by the display unit selection process (FIG. 21) of step S501, which is supplied in this manner.

[0332] The processes in steps S525 and S526 are executed in the same manner as the processes in steps S505 and S506 in Fig. 28. When the process in step S526 ends, the process returns to Fig. 19.

[0333] In this way, the display image generation unit 55 can dynamically switch the unit images to be used as display images, thereby enabling the display unit 61 to perform the same display as in the case of the camera 10. Therefore, by checking the displayed image, the user can more easily grasp the parallax between multiple images taken from different unit optical systems.

[0334] <4. Notes> <Computer> The above-described series of processes can be executed by hardware or software. When the series of processes is executed by software, the programs constituting the software are installed on a computer. Here, the term "computer" includes computers built into dedicated hardware, and general-purpose personal computers, etc., that can execute various functions by installing various programs.

[0335] FIG. 31 is a block diagram showing an example of the hardware configuration of a computer that executes the above-described series of processes by a program.

[0336] In a computer 900 shown in FIG. 31, a CPU (Central Processing Unit) 901, a ROM (Read Only Memory) 902, and a RAM (Random Access Memory) 903 are interconnected via a bus 904.

[0337] An input / output interface 910 is also connected to the bus 904. To the input / output interface 910, an input unit 911, an output unit 912, a storage unit 913, a communication unit 914, and a drive 915 are connected.

[0338] The input unit 911 includes, for example, a keyboard, a mouse, a microphone, a touch panel, an input terminal, etc. The output unit 912 includes, for example, a display, a speaker, an output terminal, etc. The storage unit 913 includes, for example, a hard disk, a RAM disk, a non-volatile memory, etc. The communication unit 914 includes, for example, a network interface. The drive 915 drives removable media 921 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

[0339] In a computer configured as above, the CPU 901 performs the above-described series of processes by, for example, loading a program stored in the storage unit 913 into the RAM 903 via the input / output interface 910 and the bus 904 and executing the program. The RAM 903 also stores data necessary for the CPU 901 to execute various processes as appropriate.

[0340] The program executed by the computer can be applied by recording it on removable media 921 such as package media, for example. In this case, the program can be installed in storage unit 913 via input / output interface 910 by inserting removable media 921 into drive 915.

[0341] This program can also be provided via a wired or wireless transmission medium such as a local area network, the Internet, digital satellite broadcasting, etc. In this case, the program can be received by the communication unit 914 and installed in the storage unit 913.

[0342] Alternatively, this program can be installed in advance in the ROM 902 or the storage unit 913 .

[0343] <Applicable targets of this technology> The present technology can be applied to any configuration. For example, the present technology can be implemented as a part of an apparatus, such as a processor as a system LSI (Large Scale Integration), a module using multiple processors, a unit using multiple modules, or a set in which other functions are added to a unit.

[0344] Furthermore, for example, the present technology can also be applied to a network system configured with multiple devices. For example, the present technology may be implemented as cloud computing in which multiple devices share and collaborate on processing via a network. For example, the present technology may be implemented in a cloud service that provides services to any terminal, such as a computer, a portable information processing terminal, or an IoT (Internet of Things) device.

[0345] In this specification, a system refers to a collection of multiple components (devices, modules (components), etc.), regardless of whether all the components are contained in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device housed in a single housing with multiple modules, are both systems.

[0346] <Fields and applications where this technology can be applied> Systems, devices, processing units, etc. to which the present technology is applied can be used in any field, for example, transportation, medical care, crime prevention, agriculture, livestock farming, mining, beauty, factories, home appliances, weather, nature monitoring, etc. In addition, their uses are also arbitrary.

[0347] <Other> The embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible without departing from the spirit of the present technology.

[0348] For example, a configuration described as one device (or processing unit) may be divided and configured as multiple devices (or processing units). Conversely, configurations described above as multiple devices (or processing units) may be combined and configured as one device (or processing unit). Of course, configurations other than those described above may be added to the configuration of each device (or each processing unit). Furthermore, as long as the configuration and operation of the entire system are substantially the same, part of the configuration of one device (or processing unit) may be included in the configuration of another device (or other processing unit).

[0349] Furthermore, for example, the above-described program may be executed in any device, as long as the device has the necessary functions (functional blocks, etc.) and is able to obtain the necessary information.

[0350] Also, for example, each step of a single flowchart may be executed by one device, or may be shared and executed by multiple devices. Furthermore, when one step includes multiple processes, the multiple processes may be executed by one device, or may be shared and executed by multiple devices. In other words, multiple processes included in one step can be executed as multiple step processes. Conversely, processes described as multiple steps can be executed collectively as one step.

[0351] For example, the steps of a program executed by a computer may be executed in chronological order in the order described herein, or may be executed in parallel or individually at the required timing, such as when a call is made. In other words, as long as no contradiction occurs, the steps may be executed in an order different from the order described above. Furthermore, the steps of this program may be executed in parallel with the processing of another program, or may be executed in combination with the processing of another program.

[0352] Furthermore, for example, multiple technologies related to the present technology can be implemented independently and independently, as long as no contradiction occurs. Of course, any multiple technologies can also be implemented in combination. For example, part or all of the present technology described in any embodiment can be implemented in combination with part or all of the present technology described in another embodiment. Furthermore, part or all of any of the above-described present technologies can be implemented in combination with other technologies not described above.

[0353] The present technology can also be configured as follows. (1) A display control unit that selectively and dynamically switches between and displays on a display unit one of a plurality of images taken from viewpoints of a plurality of unit optical systems whose optical paths are independent of each other. An imaging device comprising: (2) The display control unit selects and displays some or all of the plurality of images one by one in a predetermined order. The imaging device described in (1). (3) The display control unit selects and displays some or all of the plurality of images one by one in an order in which the scanning trajectories are line-symmetric in the relative positional relationship of the viewpoints of the plurality of images. The imaging device according to (2). (4) The display control unit selects and displays some or all of the plurality of images one by one in an order in which the scanning trajectories are rotationally symmetric in the relative positional relationship of the viewpoints of the plurality of images. An imaging device according to (2) or (3). (5) The display control unit selects, from the plurality of images, a plurality of images in which the parallax between the images is larger than the parallax between the other images, and selects and displays the selected plurality of images one by one. An imaging device according to any one of (2) to (4). (6) The display control unit selects and displays, one by one in the predetermined order, some or all of the images located on the periphery in the relative positional relationship of the viewpoints of the plurality of images. An imaging device according to any one of (2) to (5). (7) A selection order designation unit that designates a selection order of the images is further provided, The display control unit selects and displays some or all of the plurality of images one by one in the selection order designated by the selection order designation unit. An imaging device according to any one of (1) to (6). (8) The display control unit switches the selected image at predetermined intervals. An imaging device according to any one of (1) to (7). (9) The period is a single frame or multiple frames. The imaging device according to (8). (10) A period designation unit that designates the period is further provided, The display control unit switches the selected image at each of the cycles specified by the cycle specifying unit. The imaging device according to (8) or (9). (11) The image display device further includes a cutout unit that cuts out the image from a unit image having the same unit optical system as the image selected by the display control unit as a viewpoint. An imaging device according to any one of (1) to (10). (12) A region setting unit that sets a region to be cut out from the ommatidium image, The cutout unit cuts out the region set by the region setting unit from the ommatidium image as the image. The imaging device according to (11). (13) A shift amount control unit for controlling a shift amount of the position of the region is further provided, The region setting unit sets the region using the shift amount controlled by the shift amount control unit. The imaging device according to (12). (14) An imaging unit is further provided, the imaging unit corresponding to positions where the optical axes of the plurality of unit optical systems are different from each other, The display control unit selects and displays any one of the plurality of images, which are included in the image generated by the imaging unit and have each of the plurality of unit optical systems as a viewpoint, in a manner that dynamically switches the image to be displayed. An imaging device according to any one of (1) to (13). (15) The display control unit selects and displays one of the plurality of images in each frame of the captured image generated by the imaging unit. The imaging device according to (14). (16) The display control unit selects some or all of the plurality of images included in the captured image generated by the imaging unit one by one and dynamically switches and displays the images. The imaging device according to (14) or (15). (17) A plurality of imaging units corresponding to the optical axes of the plurality of unit optical systems are further provided, The display control unit selects and displays any one of the plurality of images generated by the plurality of imaging units and having each of the plurality of unit optical systems as a viewpoint, so as to dynamically switch the image to be displayed. An imaging device according to any one of (1) to (16). (18) The display control unit drives only the unit optical system corresponding to the selected image among the plurality of unit optical systems. The imaging device according to (17). (19) Selectively and dynamically switch one of a plurality of images from viewpoints of a plurality of unit optical systems whose optical paths are independent of each other and display it on a display unit. Information processing methods. (20) A computer A display control unit that selectively and dynamically switches between and displays one of a plurality of images obtained by viewing a plurality of unit optical systems having optical paths independent of each other on a display unit. A program that functions as a [Explanation of symbols]

[0354] 10 camera, 30 multi-eye optical system, 31 unitary optical system, 33 display panel unit, 34 viewfinder unit, 35 dial, 36 button, 51 image sensor, 52 RAW signal processing unit, 53 region extraction unit, 54 camera signal processing unit, 55 display image generation unit, 56 region identification unit, 57 image reconstruction processing unit, 60 bus, 61 display unit, 62 memory unit, 63 storage medium, 64 communication unit, 65 file creation unit, 70 association unit, 81 control unit, 82 memory unit, 83 storage medium, 84 optical system control unit, 141 and 142 subject, 151 unitary cut-out image, 201 display unit selection unit, 202 shift amount determination unit, 203 unitary cut-out area setting unit, 204 cut-out processing unit, 301 camera system, 310 camera body, 320 multi-lens interchangeable lens, 341 communication unit, 351 communication unit, 352 storage unit, 353 storage medium, 500 camera, 521 image selection unit

Claims

1. a display control unit that dynamically switches and displays some or all of five or more images, the five or more images having viewpoints of five or more unit optical systems whose optical paths are independent of each other, one by one in an order in which the scanning trajectories are linearly symmetric or rotationally symmetric in the relative positional relationship of the viewpoints of the five or more images; An imaging device comprising:

2. The display control unit selects, from the five or more images, a plurality of images in which the parallax between the images is larger than the parallax between the other images, and selects and displays the selected plurality of images one by one. The imaging device according to claim 1 .

3. The display control unit selects and displays part or all of the images located on the outer periphery in the relative positional relationship of the viewpoints of the five or more images one by one. The imaging device according to claim 1 .

4. a selection order designation unit that designates a selection order of the images; The display control unit selects and displays some or all of the five or more images one by one in the selection order designated by the selection order designation unit. The imaging device according to claim 1 .

5. The display control unit switches the selected image at predetermined intervals. The imaging device according to claim 1 .

6. The period may be a single frame or multiple frames. The imaging device according to claim 5 .

7. further comprising a cycle designation unit that designates the cycle, The display control unit switches the selected image at each of the cycles specified by the cycle specifying unit. The imaging device according to claim 5 .

8. a cutout unit that cuts out the image from a unit image having the same unit optical system as the image selected by the display control unit as a viewpoint thereof. The imaging device according to claim 1 .

9. further comprising a region setting unit that sets a region to be cut out from the ommatidium image, The cutout unit cuts out the region set by the region setting unit from the ommatidium image as the image. The imaging device according to claim 8 .

10. a shift amount control unit for controlling a shift amount of the position of the region; The region setting unit sets the region using the shift amount controlled by the shift amount control unit. The imaging device according to claim 9 .

11. further comprising an imaging unit corresponding to positions where the optical axes of the five or more unit optical systems are different from each other; The display control unit dynamically switches and displays some or all of the five or more images, which are included in the image generated by the imaging unit and have the five or more unit optical systems as viewpoints, one by one in an order in which a scanning trajectory becomes linearly symmetric or rotationally symmetric in a relative positional relationship between the viewpoints of the five or more images. The imaging device according to claim 1 .

12. The display control unit selects one by one and dynamically switches and displays a part or all of the five or more images in each frame of the captured image generated by the imaging unit. The imaging device according to claim 11.

13. The display control unit selects some or all of the five or more images included in the captured image generated by the imaging unit one by one and dynamically switches and displays the selected images. The imaging device according to claim 11.

14. five or more imaging units corresponding to the optical axes of the five or more unit optical systems, The display control unit dynamically switches and displays some or all of the five or more images, each of which is generated by the five or more imaging units and has as its viewpoint the five or more unit optical systems, one by one in an order in which a scanning trajectory is linearly symmetric or rotationally symmetric in a relative positional relationship between the viewpoints of the five or more images. The imaging device according to claim 1 .

15. The display control unit drives only the optical unit corresponding to the selected image among the five or more optical units. The imaging device according to claim 14.

16. A part or all of five or more images having five or more unit optical systems, each having an independent optical path as a viewpoint, are selected one by one and dynamically switched and displayed on a display unit in an order in which the scanning trajectory is linearly symmetric or rotationally symmetric in the relative positional relationship of each viewpoint of the five or more images. Information processing methods.

17. Computer, a display control unit that dynamically switches and displays some or all of five or more images, the five or more images having viewpoints of five or more unit optical systems whose optical paths are independent of each other, one by one in an order in which the scanning trajectories are linearly symmetric or rotationally symmetric in the relative positional relationship of the viewpoints of the five or more images; A program that functions as a

Images