Image processing apparatus and method, and program

Abstract

The present disclosure relates to an image processing device, method, and program that make it possible to more easily perform emulation of a lens. In the present invention, an aberration table is generated, the aberration table being of condensed light beam vectors caused by the aberration of a lens that is to be emulated and is selected via a user interface, and the aberration table and a multi-viewpoint image are used to perform a condensing process for generating a virtually captured image captured of a subject via the selected lens. The present disclosure can be applied to, for example, information processing devices, information processing devices, imaging devices, electronic equipment, information processing methods, and programs.

Description

Technical Field

[0001] The present disclosure relates to an image processing apparatus and method, and a program, and more particularly, to an image processing apparatus and method, and a program that enable easier lens emulation.

Background Art

[0002] Conventionally, it has been considered to perform virtual lens emulation to generate an image that appears to be taken with a virtual lens according to user settings from a multi-viewpoint image composed of a plurality of images with different viewpoints, and provide the image to the user (see, for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] The present disclosure enables such virtual lens emulation to be performed more easily.

Means for Solving the Problems

[0005] An image processing apparatus according to one aspect of the present technology includes A display control unit that causes a display unit to display a user interface including icons corresponding to each lens that is a candidate for emulation, and the a aberration table generation unit that generates an aberration table which is a table of converging light vectors due to aberrations of a lens to be emulated selected via a user interface, and a converging processing unit that performs converging processing to generate a virtual captured image of a subject captured through the selected lens using the aberration table and the multi-viewpoint image corresponding to the icon , using the lens parameters corresponding to the selected icon , the user interface further includes the virtual captured image generated by the light condensing process, and the light condensing processing unit performs the light condensing process in response to a drag-and-drop operation of the selected icon performed in the user interface, thereby generating the virtual captured image in which the drop position of the icon is focused and is an image processing apparatus.

[0006] ​​The image processing method of one aspect of the present technology is causes a display unit to display a user interface including icons corresponding to each lens that is a candidate for emulation, and the to generate an aberration table which is a table of converging light vectors due to aberrations of a lens to be emulated selected via a user interface corresponding to the icon , using the lens parameters corresponding to the selected icon and perform a converging process to generate a virtual captured image obtained by capturing a subject through the selected lens using the aberration table and a multi-viewpoint image in response to a drag-and-drop operation of the selected icon performed in the user interface thereby generating the virtual captured image in which the drop position of the icon is focused, and including the generated virtual captured image in the user interface This is the image processing method.

[0007] The program of one aspect of the present technology causes a computer to A display control unit that causes a display unit to display a user interface including icons corresponding to each lens that is a candidate for emulation, and the generate an aberration table which is a table of converging light vectors due to aberrations of a lens to be emulated selected via a user interface corresponding to the icon , using the lens parameters corresponding to the selected icon function as an aberration table generation unit that generates the aberration table, and a converging process unit that performs a converging process to generate a virtual captured image obtained by capturing a subject through the selected lens using the aberration table and a multi-viewpoint image , the user interface further includes the virtual captured image generated by the light condensing process, and the light condensing processing unit performs the light condensing process in response to a drag-and-drop operation of the selected icon performed in the user interface, thereby generating the virtual captured image in which the drop position of the icon is focused This is the program.

[0008] In the image processing apparatus and method, and the program of one aspect of the present technology, causes a display unit to display a user interface including icons corresponding to each lens that is a candidate for emulation, and its an aberration table which is a table of converging light vectors due to aberrations of a lens to be emulated selected via a user interface is corresponding to the icon , using the lens parameters corresponding to the selected icon generated, in response to a drag-and-drop operation of the selected icon performed in the user interface and a converging process is performed to generate a virtual captured image obtained by capturing a subject through the selected lens using the aberration table and a multi-viewpoint image thereby generating a virtual captured image in which the drop position of the icon is focused, and including the generated virtual captured image in the user interface.

Brief Description of Drawings

[0009]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Figure 23

Embodiments for Carrying Out the Invention

[0010] Hereinafter, embodiments for implementing the present disclosure (hereinafter referred to as embodiments) will be described. The description will be given in the following order. 1. Lens Emulation 2. First Embodiment (Development Processing Apparatus) 3. Supplementary Note

[0011] <1. Lens Emulation> <Multi-viewpoint Image> As an imaging device that generates a multi-viewpoint image composed of a plurality of images with different viewpoints required for lens emulation, for example, a configuration such as the camera 10 in FIG. 1 can be considered. The camera 10 incorporates an image sensor, receives light rays condensed by a lens, performs photoelectric conversion, thereby imaging a subject, and generates a multi-viewpoint image.

[0012] The camera 10 has a plurality (for example, five) of individual optical systems in a lens barrel provided on the front side (the side where light enters) of the image sensor. These plurality of individual optical systems are configured such that the optical paths of the light passing through each of them are independent of each other. That is, the light passing through each individual optical system in the lens barrel irradiates different positions on the light receiving surface (for example, the effective pixel region) of the image sensor without entering other individual optical systems. At least, the optical axes of each individual optical system are located at different places on the light receiving surface of the image sensor, and at least a part of the light passing through each individual optical system irradiates different positions on the light receiving surface of the image sensor.

[0013] Therefore, in the captured image generated by the image sensor (the entire image output by the image sensor), images of the subject formed through each individual optical system are formed at different positions. In other words, captured images with each individual optical system as the viewpoint can be obtained from the captured image.

[0014] For example, by imaging a subject with the camera 10, an overall image 20 as shown in FIG. 2 can be obtained. The overall image 20 includes individual-eye images corresponding to the respective individual-eye optical systems (images obtained by photoelectrically converting the light from the subject incident through the respective individual-eye optical systems). In the case of the example of FIG. 2, the overall image 20 includes five individual-eye images.

[0015] Note that this overall image 20 may be the entire captured image generated by the image sensor, or may be a partial image cut out from the captured image (however, including all the individual-eye images). Also, this overall image 20 may be an image in the RAW format or an image in the YC format. Note that the number of individual-eye images included in the overall image 20 is arbitrary (however, it depends on the number of individual-eye optical systems provided in the camera 10).

[0016] A part of this individual-eye image is cut out to generate a viewpoint image with the individual-eye optical system as the viewpoint. In the case of the example of FIG. 2, images can be cut out from the five viewpoint image regions 21 of the overall image 20, namely, the viewpoint image region 210, the viewpoint image region 211, the viewpoint image region 212, the viewpoint image region 213, and the viewpoint image region 214. The images cut out from these viewpoint image regions 21 are each regarded as a viewpoint image. That is, multiple viewpoint images can be obtained from the overall image 20. In other words, obtaining the overall image 20 is substantially equivalent to obtaining multiple viewpoint images.

[0017] Also, for example, by imaging a subject with the camera 10, multiple viewpoint images as shown in FIG. 3 may be obtained. In FIG. 3, the viewpoint image 220, the viewpoint image 221, the viewpoint image 222, the viewpoint image 223, and the viewpoint image 224 are viewpoint images corresponding to the respective individual-eye optical systems of the camera 10 (with the respective individual-eye optical systems as viewpoints). That is, these viewpoint images 22 can also be said to be the images cut out from the respective viewpoint image regions 21 of the above-described overall image 20. In this way, multiple viewpoint images are obtained.

[0018] Note that FIG. 3 shows an example of the viewpoint image 22 obtained by one imaging with the camera 10, but the number of this viewpoint image 22 is arbitrary (however, it depends on the number of individual optical systems provided in the camera 10).

[0019] Also, for example, by imaging a subject with the camera 10, a multi-viewpoint composite image 23 as shown in FIG. 4 may be obtained. The multi-viewpoint composite image 23 shown in FIG. 4 is an image including the plurality of viewpoint images 22 generated by synthesizing the plurality of viewpoint images 22 so as to be arranged within one image (frame). Naturally, each viewpoint image 22 can be cut out from this multi-viewpoint composite image 23. That is, a multi-viewpoint image can be obtained from the multi-viewpoint composite image 23. In other words, obtaining the multi-viewpoint composite image 23 is substantially equivalent to obtaining a multi-viewpoint image.

[0020] As described above, the camera 10 can obtain a multi-viewpoint image by imaging a subject. This multi-viewpoint image can be used, for example, for generating depth information and processing such as refocusing using the depth information.

[0021] Note that the method for generating the multi-viewpoint image is arbitrary, and it may be generated using a device or system other than the camera 10. For example, a multi-viewpoint image may be generated by imaging a subject using a plurality of cameras (a plurality of image sensors) having different viewpoints from each other.

[0022] <Utilization of multi-viewpoint image> For example, by using a method such as stereo matching, depth data indicating the distance to the subject in the multi-viewpoint image is obtained from a multi-viewpoint image composed of a plurality of viewpoint images having different viewpoints from each other, and each viewpoint image constituting the multi-viewpoint image is overlapped using the depth data to generate an image, and the image can be provided to the user.

[0023] Using such a method, for example, a depth image 31 as shown in A of FIG. 5 can be generated from a multi-viewpoint image. This depth image 31 is an image having the distance to the subject at each pixel of the captured image as a pixel value. That is, the depth image 31 is an image obtained by imaging depth data. Further, by superimposing (superimposing and synthesizing) each viewpoint image constituting the multi-viewpoint image at a ratio determined using the depth image 31, a virtual captured image 32 as shown in B of FIG. 5 can be generated.

[0024] By such a method, for example, a virtual captured image 32 can be generated by arbitrarily setting the focal length. That is, a virtual captured image 32 can be generated by arbitrarily setting lens parameters which are parameters related to the lens. In other words, a virtual lens can be set, and a virtual captured image can be generated which emulates a captured image obtained by imaging a subject through the virtual lens. That is, lens emulation can be performed.

[0025] For example, a virtual lens 40 as shown in FIG. 6 is set, and an imaging system including a ray vector 41 which is incident light from the subject incident on the virtual lens 40, a ray vector 42 which is the incident light after passing through the virtual lens 40, and an image sensor 43 which obtains a captured image from the ray vector 42 is emulated (simulatedly manifested). Then, based on the emulation, a virtual captured image is generated from the multi-viewpoint image.

[0026] In such lens emulation, an optical system with any specifications can be emulated. That is, the object of emulation may be an existing optical system or an unrealistic optical system. Through such emulation, it is possible to reproduce the imaging images obtained by various optical systems. For example, an imaging image obtained using an optical system with a larger aperture than the optical system used for generating a multi-viewpoint image, an imaging image with a higher resolution than the multi-viewpoint image, an imaging image with a wider angle of view than the multi-viewpoint image, an imaging image with a narrower angle of view and a longer focal length than the multi-viewpoint image, etc. can also be easily reproduced. Therefore, for example, an imaging image captured using a high-class lens or an imaging image captured using an unrealistic optical system can also be easily reproduced. That is, a wider variety of imaging images (virtual imaging images) can be generated at a lower cost.

[0027] <Aberration setting by lens selection> In the present disclosure, an aberration table, which is a table of converging light vectors based on the aberration of the lens to be emulated selected via a user interface, is generated, and a converging process is performed to generate a virtual imaging image of an object captured through the selected lens using the aberration table and the multi-viewpoint image.

[0028] By doing so, the work of setting parameters related to aberration becomes unnecessary, so that lens emulation can be performed more easily.

[0029] Note that the imaging image generated by the converging process is an image obtained by virtual imaging using the selected lens. Therefore, this imaging image can also be said to be a virtual imaging image. Also, the lens selected here does not have to be the one used for actual imaging, and it may exist in the real world or may not exist.

[0030] <Development screen> The lens emulation using the multi-viewpoint image as described above may be realized by an application program that performs processing based on user operation inputs. For example, the application program may display a GUI (Graphical User Interface) on a monitor and accept selection of a lens to be emulated by the user.

[0031] For example, a user interface may be displayed on a display unit, and the user interface includes icons corresponding to each lens that is a candidate for emulation, and a aberration table may be generated using the lens parameters corresponding to the icon selected via the user interface.

[0032] The development screen 100 shown in FIG. 7 is a GUI for performing development processing of a multi-viewpoint image, which is displayed on a monitor by an application program. This application program can perform, as the development processing of the multi-viewpoint image, for example, lens emulation using the multi-viewpoint image as described above, and generate a virtual captured image obtained by capturing a subject through the lens. At this time, the application program causes a development screen 100 as shown in FIG. 7 to be displayed on the monitor and accepts a user operation performed based on the development screen 100.

[0033] As shown in FIG. 7, the development screen 100 has a tab display unit 101 and a GUI display unit 102. Tabs for switching the GUI displayed on the GUI display unit 102 are displayed on the tab display unit 101. In the case of the example of FIG. 7, tabs 101A (Tab1), 101B (Tab2), 101C (CGLE), and 101D (Gen_onj) are displayed on the tab display unit 101. The number of tabs displayed on the tab display unit 101 is arbitrary.

[0034] In the GUI display unit 102, a GUI corresponding to the tab selected in the tab display unit 101 is displayed. In the example of FIG. 7, tab 101A is selected, and the GUI corresponding to tab 101A is displayed in the GUI display unit 102. This GUI is a GUI for lens emulation.

[0035] As shown in FIG. 7, this GUI includes an imaging image display unit 110, an input / output setting unit 111, a depth estimation unit 112, and a lens emulation unit 113.

[0036] The imaging image display unit 110 can display, as a light condensing processing result, an imaging image (virtual imaging image) obtained by imaging using an emulated lens generated by light condensing processing. By checking the imaging image displayed on this imaging image display unit 110, the user can grasp the light condensing processing result.

[0037] Note that the lens emulation can be performed using a low-resolution multi-viewpoint image. In this case, the virtual imaging image obtained as an emulation result has a lower resolution than the original multi-viewpoint image (the multi-viewpoint image before being low-resolutionized). That is, the low-resolution virtual imaging image is displayed on the imaging image display unit 110. In other words, the imaging image display unit 110 can display a virtual imaging image with a lower resolution than the original multi-viewpoint image obtained by actual imaging. By doing so, an increase in the load due to the light condensing processing and the display of its light condensing processing result can be suppressed. As a result, the light condensing processing and the display of its light condensing processing result can be performed faster without increasing the cost. Therefore, the imaging image display unit 110 can, for example, perform the display of the light condensing processing result faster. Of course, the original multi-viewpoint image may be used for the light condensing processing. In this case, as the light condensing processing result, a virtual imaging image with the same resolution as the original multi-viewpoint image is obtained. That is, the imaging image display unit 110 may display a virtual imaging image with the same resolution as the original multi-viewpoint image.

[0038] The input / output setting unit 111 is a GUI for inputting information related to input / output settings. The depth estimation unit 112 is a GUI for depth estimation. The lens emulation unit 113 is a GUI for lens emulation. This lens emulation unit 113 includes a lens selection unit 121 and a lens setting unit 122. The lens selection unit 121 is a GUI for selecting a lens to be emulated. The lens setting unit 122 is a GUI for lens settings. The configuration of each GUI will be described below.

[0039] <Input / Output Setting Unit> FIG. 8 is a diagram showing an example of the input / output setting unit 111. The input / output setting unit 111 is a GUI for inputting information related to input / output settings. The information input in this input / output setting unit 111 is received by the application program and used for input / output settings.

[0040] For example, the input / output setting unit 111 has an input file specification field 131A and a reference button 131B. The input file specification field 131A is a GUI for setting a file to be input (also referred to as an input file). This input file includes, for example, data of a multi-viewpoint image used for condensing processing. That is, by specifying this input file, the multi-viewpoint image to be input is specified. The user can operate, for example, a keyboard or the like to input a file name or address into this input file specification field 131A using characters or symbols. The file specified by the file name or address input in this input file specification field 131A is set as the input file. That is, the data of the multi-viewpoint image included in the input file is read by the application program.

[0041] The reference button 131B is a GUI button for specifying an input file using a file management GUI. When the user operates this reference button 131B, the file management GUI is launched. The user specifies a file using the file management GUI. When a file is specified, the file name and address of the file are input into the input file specification field 131A. That is, the file is set as the input file. That is, data such as multi-viewpoint image data included in the input file is read.

[0042] Also, the input / output setting unit 111 has an output folder specification field 132A and a reference button 132B. The output folder specification field 132A is a GUI for setting a folder (also referred to as an output folder) that outputs files including data such as captured images of the light condensing processing results. The user can input a folder name or address into the output folder specification field 132A using characters or symbols by operating, for example, a keyboard. The folder specified by the folder name or address input into the output folder specification field 132A is set as the output folder. That is, the generated file is stored in the output folder.

[0043] The reference button 132B is a GUI button for specifying an output folder using a file management GUI. When the user operates this reference button 132B, the file management GUI is launched. The user specifies a folder using the file management GUI. When a folder is specified, the folder name and address of the folder are input into the output folder specification field 132A. That is, the folder is set as the output folder. That is, the generated file is stored in the output folder.

[0044] Furthermore, the input / output setting unit 111 has a calibration file designation field 133A and a reference button 133B. The calibration file designation field 133A is a GUI for setting a file (also referred to as a calibration file) in which calibration data and the like for specifying distortion information between viewpoints are described. That is, by designating this calibration file, calibration data used for calibration between viewpoints is set. The user can input a file name or address into this calibration file designation field 133A using characters or symbols by operating, for example, a keyboard or the like. The file specified by the file name or address input into this calibration file designation field 133A is set as the calibration file. That is, the calibration data and the like included in the calibration file are read.

[0045] The reference button 133B is a GUI button for designating a calibration file using a GUI for file management. When the user operates this reference button 133B, the GUI for file management is activated. The user designates a file using the GUI for file management. When a file is designated, the file name and address of the file are input into the calibration file designation field 133A. That is, the file is set as the calibration file. That is, the calibration data and the like included in the calibration file are read.

[0046] Table 151 shown in FIG. 9 shows an example of information described in the calibration file. For example, the model mode is information indicating the mode of the camera 10. The baseline length is information indicating the distance between each individual optical system. CMN is information indicating the number of individual optical systems (i.e., the number of viewpoints). The DepthMax initial value is information indicating the initial value of the maximum value (DepthMax) of the depth value described later. The EDGE initial value is information indicating the initial value of EDGE described later.

[0047] Raw_width is information indicating the width (horizontal resolution) of the entire captured image obtained by the image sensor of the camera 10, i.e., the RAW image. Raw_height is information indicating the height (vertical resolution) of the entire captured image. Pros_width is information indicating the width (horizontal resolution) of each viewpoint image. Pros_height is information indicating the height (vertical resolution) of each viewpoint image.

[0048] CentrePros[N].x (where N is an integer from 0 to 4, for example) indicates the x-coordinate of the center of each viewpoint image in the entire captured image obtained by the image sensor of the camera 10, and CentrePros[N].y (where N is an integer from 0 to 4, for example) indicates the y-coordinate thereof. Note that N indicates the identification number of the viewpoint image (individual optical system). For example, in the case of the camera 10 in FIG. 1, since the number of viewpoints is 5, any one of 0, 1, 2, 3, 4 is set for N.

[0049] The horizontal width [mm] of the calibration chart is information indicating the horizontal width of the chart used for calibration (also referred to as the calibration chart) (see, for example, International Publication No. 2019 / 078032) in length [mm]. The horizontal width [pixel] of the calibration chart is information indicating the horizontal width at the resolution [pixel]. Further, the distance [mm] from the camera to the calibration chart is information indicating the distance [mm] from the camera 10 to the calibration chart during calibration imaging (i.e., the depth value of the calibration chart).

[0050] These pieces of information are set when the calibration file is specified. Of course, the content of the calibration file is arbitrary, and it may contain calibration data other than the example shown in FIG. 9, or may not contain a part of the calibration data shown in FIG. 9.

[0051] Returning to FIG. 8, the input / output setting unit 111 has a development setting file specification field 134A and a reference button 134B. The development setting file specification field 134A is a GUI for setting a file (also referred to as a development setting file) in which hardware information such as the number of nozzles and the baseline length is described. The user can operate, for example, a keyboard or the like to input a file name or address into the development setting file specification field 134A using characters and symbols. The file specified by the file name or address input into the development setting file specification field 134A is set as the development setting file. That is, the hardware information and the like included in the development setting file are read.

[0052] The reference button 134B is a GUI button for specifying a development setting file using a GUI for file management. When the user operates the reference button 134B, the GUI for file management is launched. The user specifies a file using the GUI for file management. When a file is specified, the file name and address of the file are input into the development setting file specification field 134A. That is, the file is set as the development setting file. That is, the hardware information and the like included in the development setting file are read.

[0053] Table 152 shown in FIG. 10 shows an example of hardware information described in the development setting file. The development setting file includes various hardware information such as, for example, model information, baseline length, number of viewpoints, calibration adjustment value, initial DepthMax value, initial EDGE value, processing margin, resolution, processing resolution, provisional value, color filter information, viewpoint N center coordinates, focal length, LED coordinates indicating the coordinates of the spot light for detecting lens mounting error (see, for example, International Publication No. 2019 / 78032), LED processing resolution indicating the processing resolution of the spot light (see, for example, International Publication No. 2019 / 78032), information at the time of calibration, etc. These pieces of information are set when the development setting file is specified. Of course, the content of the development setting file is arbitrary, and for example, it may include hardware information other than the example shown in FIG. 10, or may not include some of the hardware information shown in FIG. 10.

[0054] Returning to FIG. 8, the input / output setting unit 111 has an HDR (High Dynamic Range) file designation field 135A and a reference button 135B. The HDR file designation field 135A is a GUI for setting a file (also referred to as an HDR file) that enables estimation of the saturated luminance amount. The user can input a file name or address into this HDR file designation field 135A using characters or symbols by operating, for example, a keyboard or the like. The file specified by the file name or address input into this HDR file designation field 135A is set as the HDR file. That is, the information contained in the HDR file is read.

[0055] The reference button 135B is a GUI button for specifying an HDR file using the GUI for file management. When the user operates this reference button 135B, the GUI for file management is activated. The user specifies a file using the GUI for file management. When a file is specified, the file name and address of the file are input into the HDR file designation field 135A. That is, the file is set as the HDR file. That is, the information contained in the HDR file is read.

[0056] In addition, the input / output setting unit 111 has a threshold value specification field 136, a gain specification field 137, and an HDR button 138.

[0057] The threshold value specification field 136 is a GUI for setting a threshold value (TH). The user can input a threshold value into this threshold value specification field 136 by operating, for example, a keyboard or the like. The numerical value input into this threshold value specification field 136 is set as the threshold value.

[0058] The gain specification field 137 is a GUI for setting a gain. The user can input a gain numerically into this gain specification field 137 by operating, for example, a keyboard or the like. The numerical value input into this gain specification field 137 is set as the gain.

[0059] The HDR button 138 is a GUI for starting HDR processing. When the user operates this HDR button 138, HDR processing is started. At that time, the HDR file specified in the HDR file specification field 135A is used for the processing. Note that, for example, the HDR file and HDR processing disclosed in International Publication No. 2016 / 175043 can be used.

[0060] <Depth Estimation Unit> FIG. 11 is a diagram showing an example of the depth estimation unit 112. The depth estimation unit 112 is a GUI for inputting information related to the estimation of depth. The information input in this depth estimation unit 112 is received by the application program and used for depth estimation.

[0061] For example, the depth estimation unit 112 has a depth image (depth map) display field 161 and an operation button 162. The depth image display field 161 is a GUI for displaying a depth image composed of the depth values of the subjects of each pixel as an estimation result of depth estimation. That is, the depth image displayed in this depth image display field 161 is composed of the depth values estimated by depth estimation.

[0062] The calculation button 162 is a GUI for instructing the start of its depth estimation (depth calculation). When the user operates this calculation button 162, the depth estimation is started.

[0063] In addition, the depth estimation unit 112 has a depth file specification column 163A and a reference button 163B. The depth file specification column 163A is a GUI for setting a file (also referred to as a depth file) to be used when externally inputting depth data. The user can, for example, operate a keyboard or the like to input a file name or address into this depth file specification column 163A using characters and symbols. The file specified by the file name or address input into this depth file specification column 163A is set as the depth file. That is, the depth data and the like included in the depth file are read.

[0064] The reference button 163B is a GUI button for specifying a depth file using a GUI for file management. When the user operates this reference button 163B, the GUI for file management is launched. The user specifies a file using the GUI for file management. When a file is specified, the file name and address of the file are input into the depth file specification column 163A. That is, the file is set as the depth file. That is, the depth data and the like included in the depth file are read.

[0065] In addition, the depth estimation unit 112 has an EDGE specification column 165, a MAX specification column 166, a CS specification column 167, a gamma specification column 168, and an ALE specification column 169. The EDGE specification column 165 is a GUI for specifying an EDGE threshold setting for separating noise and signals using the values of the shutter speed / gain at the time of shooting. The user can, for example, operate a keyboard or the like to input a numerical value into this EDGE specification column 165. The numerical value input into this EDGE specification column 165 is set as the EDGE threshold.

[0066] The MAX specification field 166 is a GUI for specifying the detection range of the maximum pixel displacement amount. The user can input a numerical value into this MAX specification field 166 by operating, for example, a keyboard or the like. The numerical value input into this MAX specification field 166 is set as the detection range MAX of the maximum pixel displacement amount.

[0067] The CS specification field 167 is a GUI for specifying infinity during calibration. The user can input a numerical value into this CS specification field 167 by operating, for example, a keyboard or the like. The numerical value input into this CS specification field 167 is set as infinity (CS) during calibration.

[0068] The gamma specification field 168 is a GUI for specifying the gamma curve to be used during depth estimation. The user can input a numerical value into this gamma specification field 168 by operating, for example, a keyboard or the like. The numerical value input into this gamma specification field 168 is set as the gamma value.

[0069] The ALE specification field 169 is a GUI for specifying the rotation angle ALE that can be corrected in the rotation direction during calibration data shooting. The user can input a numerical value into this ALE specification field 169 by operating, for example, a keyboard or the like. The numerical value input into this ALE specification field 169 is set as the rotation angle ALE that can be corrected in the rotation direction during calibration data shooting.

[0070] <Lens selection section> FIG. 12 is a diagram showing an example of the lens selection section 121. The lens selection section 121 is a GUI for specifying the lens to be emulated. The information input in this lens selection section 121 is received by the application program and used for lens emulation.

[0071] For example, the lens selection section 121 includes a lens icon display section 171, an add button 172, a delete button 173, and a processing result storage section 174.

[0072] The lens icon display unit 171 is a GUI for selecting a lens to be emulated, and includes icons corresponding to each lens that is a candidate for emulation. In the case of the example in FIG. 12, lens icons 181-1 to lens icons 181-4 are displayed. When there is no need to distinguish and describe the lens icons 181-1 to lens icons 181-4 from each other, they are referred to as lens icon 181.

[0073] The lens icons 181-1 to lens icons 181-4 are icons corresponding to different lenses, and different lens parameters are associated with each lens icon 181. For example, when the user selects any one of the lens icons 181, the lens corresponding to the selected lens icon 181 is selected (set) as the object of emulation. That is, the lens parameter of the value corresponding to the selected lens icon 181 is set.

[0074] Then, using the set parameters, emulation of the lens is performed, aberration is set, and light condensing processing is performed, and the light condensing processing result (captured image) is displayed on the captured image display unit 110. In this way, the application program can emulate the lens corresponding to the selected lens icon 181.

[0075] For the emulation of the lens, setting of a large number of parameters is required. By using the selection of the lens icon in this way, the user or the like can more easily perform the emulation of the lens.

[0076] In particular, the lens parameters associated with the lens icon 181 include parameters for specifying aberration, and the user or the like can specify aberration by selecting the lens icon 181.

[0077] Incidentally, as described above, in the lens icon display unit 171, lens icons 181 corresponding to lenses that are candidates for emulation are displayed. That is, in the lens icon display unit 171, a list of lenses that are candidates for emulation is displayed. Therefore, by referring to this lens icon display unit 171, a user or the like can easily grasp the candidates for the emulation target.

[0078] Note that the lens icons 181-1 to 181-3 are icons corresponding to lenses prepared in advance, that is, icons corresponding to lenses for which lens parameters are set in advance (preset lens icons) (test Lens). That is, in the lens icon display unit 171, such lens icons 181 corresponding to lenses prepared in advance can be displayed without the user or the like generating a lens icon. Therefore, a user or the like can select a lens to be emulated without the need for complicated operations such as generating a lens icon.

[0079] On the other hand, the lens icon 181-4 is an icon corresponding to a lens generated by a user or the like, that is, an icon corresponding to a lens for which lens parameters are set by a user or the like (user setting). That is, in the lens icon display unit 171, such lens icons 181 corresponding to lenses generated by a user or the like can be displayed. Therefore, a user or the like can select lenses with more diverse specifications as targets for emulation.

[0080] Also, a user or the like may be able to select a subject to be focused on by moving the lens icon 181 to the imaging image display unit 110 by a drag-and-drop operation.

[0081] For example, when a user or the like drags and drops the lens icon 181 displayed on the lens icon display unit 171 onto the captured image display unit 110, the lens corresponding to the dragged and dropped lens icon 181 is set as the lens to be emulated. Further, the position where the lens icon 181 is moved and dropped on the captured image displayed on the captured image display unit 110 is set as the focusing position. That is, lens emulation is performed using the lens parameters corresponding to the dragged and dropped lens icon 181, aberration is set, and light collection processing is performed so as to focus on the dropped position of the lens icon 181, and the light collection processing result (captured image) is displayed on the captured image display unit 110. By doing so, the user or the like can more easily set not only the lens to be emulated but also the focusing position.

[0082] Note that the focusing position may be a region composed of a plurality of pixels centered on the dropped position. Also, the subject existing at the dropped position may be recognized by image recognition, and focusing may be performed on the subject. Further, the number of lens icons 181 displayed on the lens icon display unit 171 is arbitrary and is not limited to the example of FIG. 12.

[0083] The add button 172 is a GUI button for adding a new lens icon 181 to the lens icon display unit 171. For example, when a user or the like operates this add button 172, a lens icon 181 associated with the lens parameters set in the lens setting unit 122 described later is generated and displayed on the lens icon display unit 171. That is, a new lens icon 181 is added to the lens icon display unit 171.

[0084] The delete button 173 is a GUI button for deleting the lens icon 181 displayed on the lens icon display section 171. For example, when a user or the like operates this delete button 173, the lens icon 181 selected by the user or the like among the lens icons 181 displayed on the lens icon display section 171 is deleted. As a result, the lens corresponding to the lens icon 181 cannot be selected as an object of emulation. Note that the lens icon 181 that can be deleted may be limited to the lens icon 181 added by a user or the like. In other words, the preset lens icon 181 may be made undeletable.

[0085] The processing result storage section 174 includes a GUI for setting information regarding the storage of the emulation result (condensing processing result), that is, the virtual captured image displayed on the captured image display section 110. For example, the processing result storage section 174 has an image quality selection section 191, a save button 192, and a focus range setting section 193. The image quality selection section 191 is a GUI that enables a user to select the image quality (resolution) when saving the virtual captured image. For example, the image quality selection section 191 has radio buttons with "normal image quality" and "low image quality" as options, and the user can select the image quality (resolution) of the virtual captured image at the time of saving by operating this image quality selection section 191. That is, the virtual captured image is saved with the image quality (resolution) selected by the user.

[0086] As described above, for reducing the load, a low-resolution multi-viewpoint image can be used for emulation of a lens. In that case, the virtual captured image displayed on the captured image display section 110 has a lower resolution than the captured image. The user can select whether to save such a virtual captured image at the resolution at the time of emulation or at a resolution equivalent to that of the original multi-viewpoint image by operating the image quality selection section 191.

[0087] For example, when the user selects "low image quality", the virtual captured image is saved at the resolution used for lens emulation. That is, the virtual captured image (emulation result) displayed on the captured image display unit 110 is saved as is at that resolution. Therefore, in this case, the virtual captured image can be saved with a lower load (faster).

[0088] On the other hand, when the user selects "normal image quality", the virtual captured image is saved at a resolution equivalent to the original multi-viewpoint image. In this case, the emulation is reproduced using the original multi-viewpoint image. That is, a virtual captured image similar to the virtual captured image displayed on the captured image display unit 110 (that is, a virtual captured image equivalent to the virtual captured image displayed on the captured image display unit 110 with increased resolution) is generated and saved, except for the resolution.

[0089] Here, as an example of the options for the image quality (resolution) at the time of saving, two types are shown: the resolution at the time of emulation (low image quality) and the resolution equivalent to the original multi-viewpoint image (normal image quality). However, the options for this image quality (resolution) are arbitrary and are not limited to the above example. For example, the virtual captured image may be saved at a resolution (image quality) that is lower than the original multi-viewpoint image and higher than the resolution at the time of emulation. That is, also, the virtual captured image may be saved at a resolution lower than the resolution at the time of emulation. That is, the virtual captured image can be saved at any resolution (image quality).

[0090] Note that when saving, the resolution (image quality) of the virtual captured image may be converted to the desired resolution (image quality). Also, in the above, the virtual captured image has been described, but the depth image can also be saved at any resolution (image quality) in the same way.

[0091] The save button 192 is a GUI button for executing saving. For example, when the user or the like operates this save button 192, the virtual captured image (condensing process result) displayed on the captured image display unit 110 is saved in a storage medium or the like with the image quality selected by the image quality selection unit 191. Also, the focus range setting unit 193 is a GUI for inputting information for designating the focal length.

[0092] <Lens setting unit> FIG. 13 is a diagram showing an example of the lens setting unit 122. The lens setting unit 122 is a GUI for setting (customizing) lens parameters. The information input in this lens setting unit 122 is received by the application program and used for lens emulation.

[0093] The lens setting unit 122 has a simple lens emulation result display unit 201, an image height input field 202A, an image height setting slide bar 202B, a focus position input field 202C, and a focus position setting slide bar 202D.

[0094] The simple lens emulation result display unit 201 can display a virtual captured image with a point light source as the subject, representing the result of the condensing process. For example, the simple lens emulation result display unit 201 performs real-time result display of how the point light source located at the horizontal image height set using the image height input field 202A and the image height setting slide bar 202B is blurred at the distance from the focus plane set using the focus position input field 202C and the focus position setting slide bar 202D. By doing so, the user or the like can visually (intuitively) understand the effects of various aberrations specified by the GUI described later.

[0095] The lens setting unit 122 has, as a GUI for performing settings related to the aperture, an aperture input field 203A for inputting the aperture value inside the lens, and an aperture display section 203B for displaying the state of the set aperture. Also, the lens setting unit 122 has, as a GUI for performing settings related to the focus position, a focus position input field 204A for numerically inputting the focus position of the lens, and a focus position slide bar 204B for setting the focus position by the position of a knob.

[0096] In addition, the lens setting unit 122 has a GUI for performing settings related to aberrations. For example, the lens setting unit 122 has, as such a GUI, a GUI for setting astigmatism, a GUI for setting spherical aberration, a GUI for setting coma aberration, a GUI for setting distortion aberration, and a GUI for setting field curvature.

[0097] For example, the lens setting unit 122 has, as a GUI for setting astigmatism, an astigmatism coefficient input field 205A for numerically inputting the astigmatism coefficient, and an astigmatism coefficient slide bar 205B for setting the astigmatism coefficient by the position of a knob. Also for example, the lens setting unit 122 has, as a GUI for setting spherical aberration, a spherical aberration coefficient input field 206A for numerically inputting the spherical aberration coefficient, and a spherical aberration coefficient slide bar 206B for setting the spherical aberration coefficient by the position of a knob. Further for example, the lens setting unit 122 has, as a GUI for setting coma aberration, a coma aberration coefficient input field 207A for numerically inputting the coma aberration coefficient, and a coma aberration coefficient slide bar 207B for setting the coma aberration coefficient by the position of a knob. Also for example, the lens setting unit 122 has, as a GUI for setting distortion aberration, a distortion aberration coefficient input field 208A for numerically inputting the distortion aberration coefficient, and a distortion aberration coefficient slide bar 208B for setting the distortion aberration coefficient by the position of a knob. Further for example, the lens setting unit 122 has, as a GUI for setting field curvature, a field curvature coefficient input field 209A for numerically inputting the field curvature coefficient, and a field curvature coefficient slide bar 209B for setting the field curvature coefficient by the position of a knob.

[0098] These various aberrations are called the five Seidel aberrations.

[0099] In addition, the lens setting unit 122 includes a load button 210, which is a GUI button for loading lens settings from a predetermined file when the user operates it, and a write button 211, which is a GUI button for writing lens settings to a predetermined file when the user operates it.

[0100] Furthermore, when using an external focus map, the lens setting unit 122 includes a focus map input field 212A for specifying the focus map and a reference button 212B. The focus map is map information in which parallax information is registered for each pixel (position) of the captured image.

[0101] When at least any one of the settings related to the aperture, the settings related to the focus position, and the settings related to the aberration change due to operations by the user or the like on the above GUI, the light collection process is immediately (in real time) executed, and a processing result (virtual captured image) reflecting the changes in those parameters is displayed on the captured image display unit 110.

[0102] Therefore, it is possible to more easily perform lens emulation.

[0103] <Other image synthesis process> Also, an other image synthesis process may be performed to synthesize the virtual captured image, which is the light collection process result, and a synthesis target image, which is another image such as a CG image. In this case, a synthesis process (depth-corresponding synthesis process) that takes into account the depth data set in the synthesis target image and the depth data of the virtual captured image may be performed. FIG. 14 is a diagram showing an example of the development screen 100 when performing such an other image synthesis process. For example, when the user or the like selects the tab 101C in the tab display unit 101 of the development screen 100, a GUI for the other image synthesis process corresponding to the tab 101C is displayed on the GUI display unit 102.

[0104] As shown in FIG. 14, this GUI has a composite image display section 310, an input / output setting section 311, and a lens emulation section 313. The lens emulation section 313 has a lens selection section 321 and a lens setting section 322.

[0105] The composite image display section 310 can display a composite image obtained by synthesizing an imaging image (virtual imaging image) obtained by imaging using an emulated lens generated by a condensing process and a synthesis target image. Also, a user or the like can perform an operation for synthesizing the synthesis target image with the virtual imaging image in the composite image display section 310.

[0106] Similar to the imaging image display section 110, the composite image display section 310 can display a virtual imaging image that is a result of the condensing process. Then, a user or the like can perform an operation for synthesizing the synthesis target image 310A with the virtual imaging image.

[0107] Note that, similar to the case of the imaging image display section 110, the composite image display section 310 can display a virtual imaging image with a lower resolution than the original multi-viewpoint image obtained by actual imaging. By doing so, an increase in load due to the condensing process, display of the condensing process result, synthesis of the synthesis target image, etc. can be suppressed. As a result, for example, processes such as the condensing process, display of the condensing process result, and display of the composite image can be performed at a higher speed without increasing the cost. Of course, the composite image display section 310 may display a virtual imaging image with the same resolution as the original multi-viewpoint image. That is, the original multi-viewpoint image may be used for the condensing process, and a virtual imaging image with the same resolution as the original multi-viewpoint image may be obtained as a result of the condensing process.

[0108] <Input / Output Setting Section> The input / output setting unit 311 is a GUI for inputting information regarding input / output settings, similar to the input / output setting unit 111. The information input in this input / output setting unit 311 is received by the application program and used for input / output settings. An example of this input / output setting unit 311 is shown in FIG. 15.

[0109] For example, the input / output setting unit 311 has a depth file specification field 331A and a reference button 331B, and is a GUI for specifying a depth file including depth data. For example, a user or the like inputs a file name or address in the depth file specification field 331A, or operates the reference button 331B to start a GUI for file management, and uses the GUI for file management to specify a file, thereby specifying the depth file to be input.

[0110] Also, the input / output setting unit 311 has an RGB file specification field 332A and a reference button 332B, and is a GUI for specifying an RGB file including an RGB image. For example, a user or the like inputs a file name or address in the RGB file specification field 332A, or operates the reference button 332B to start a GUI for file management, and uses the GUI for file management to specify a file, thereby specifying the RGB file to be input.

[0111] Note that this RGB image is a so-called captured image (RAW image), and is an image corresponding to RGB data composed of pixel values of R pixels, G pixels, and B pixels. On the other hand, the depth image is obtained by imaging depth data corresponding to the RGB image by assigning colors to each depth value. In other words, the depth image is map information that represents the distance (depth value) to the subject at each pixel of the RGB image by the color (pixel value) of each pixel.

[0112] Note that the RGB image does not have to be a RAW image in a Bayer array. Also, the color space of the captured image is arbitrary and does not have to be an RGB image. For example, it may be an image such as YUV, YCbCr, YPbPr, etc. Furthermore, the captured image does not have to be a color image. For example, it may be a monochromatic image such as a monochrome image or a sepia-toned image. That is, this RGB image may be anything as long as it takes the form of an image.

[0113] Furthermore, the input / output setting unit 311 has a PSF file specification column 333A and a reference button 333B, which are GUIs for specifying a PSF (Point Spread Function) file including information on the point spread function. For example, a user or the like inputs a file name or address in the PSF file specification column 333A, or operates the reference button 333B to activate a GUI for file management, and specifies a file using the GUI for file management to specify the input PSF file.

[0114] Also, the input / output setting unit 311 has an output folder specification column 334A and a reference button 334B, which are GUIs for specifying a folder for outputting a file of the composite image that is the result of other image composite processing. For example, a user or the like inputs a folder name or address in the output folder specification column 334A, or operates the reference button 334B to activate a GUI for file management, and specifies a folder using the GUI for file management to specify a folder for outputting a file of the composite image. Furthermore, the input / output setting unit 311 has an FPSx2 input column 335 for inputting the frame rate.

[0115] <Lens selection unit> Returning to FIG. 14, the lens emulation unit 313 is a GUI related to lens emulation. The lens selection unit 321 within the lens emulation unit 313 is a GUI related to the selection of the lens to be emulated. An example of this lens selection unit 321 is shown in FIG. 16.

[0116] As shown in FIG. 16, for example, the lens selection unit 321 includes a lens icon display unit 341 and a processing result storage unit 342. The lens icon display unit 341 is a GUI for selecting a lens to be emulated, similar to the lens icon display unit 171, and icons corresponding to lenses that are candidates for emulation are displayed. In the example of FIG. 16, lens icons 351-1 to 351-4 are displayed. When there is no need to distinguish and describe the lens icons 351-1 to 351-4 from each other, they are referred to as lens icon 351.

[0117] The lens icons 351-1 to 351-4 are icons similar to the lens icons 181-1 to 181-4, respectively. Similar to the case of the lens icon display unit 171, for example, when the user selects any of the lens icons 351, the lens corresponding to the selected lens icon 351 is selected (set) as the object of emulation. That is, the lens parameters of the value corresponding to the selected lens icon 351 are set. Then, using the set parameters, lens emulation is performed, aberration is set, and light collection processing is performed, and the light collection processing result (captured image) is displayed on the composite image display unit 310.

[0118] That is, also in the GUI for other image composition processing corresponding to tab 101C, the user or the like can more easily perform lens emulation. In particular, the lens parameters associated with the lens icon 351 include parameters for specifying aberration, and the user or the like can specify aberration by selecting the lens icon 181.

[0119] Incidentally, as described above, the lens icon display unit 341 displays the lens icon 351 corresponding to the lens that is a candidate for emulation. Therefore, the user or the like can easily grasp the candidates for emulation by referring to this lens icon display unit 341.

[0120] Note that lens icons 351-1 to 351-3 are icons corresponding to lenses prepared in advance, that is, icons corresponding to lenses with lens parameters set in advance (preset lens icons) (test Lens). Therefore, users and the like can select a lens to be emulated without the need for complicated operations such as generating lens icons.

[0121] On the other hand, lens icon 351-4 is an icon corresponding to a lens generated by a user or the like, that is, an icon corresponding to a lens with lens parameters set by a user or the like (user setting). Therefore, users and the like can select lenses with more diverse specifications as objects of emulation.

[0122] Also, similar to the case of the lens icon display unit 171, users and the like may be able to select a subject to be focused by dragging and dropping the lens icon 351 displayed on the lens icon display unit 341 onto the composite image display unit 310. That is, lens emulation may be performed using the lens parameters corresponding to the dragged and dropped lens icon 351, aberrations may be set, and light collection processing may be performed so as to focus on the subject located at the destination of movement of the lens icon 351, and the light collection processing result (captured image) may be displayed on the composite image display unit 310.

[0123] Note that the number of lens icons 351 displayed on the lens icon display unit 341 is arbitrary and is not limited to the example in FIG. 16.

[0124] The processing result storage unit 342 has the same GUI as the processing result storage unit 174. The processing result storage unit 342 includes an image quality selection unit 361, a save button 362, and a focus range setting unit 363. The image quality selection unit 361 has the same GUI as the image quality selection unit 191. The save button 362 has the same GUI as the save button 192. The focus range setting unit 363 has the same GUI as the focus range setting unit 193.

[0125] <Lens setting section> Returning to FIG. 14, the lens setting section 322 is a GUI for setting (customizing) lens parameters. The information input in this lens setting section 322 is received by the application program and used for lens emulation. An example of this lens setting section 322 is shown in FIG. 17.

[0126] As shown in FIG. 17, the lens setting section 322 has a simple lens emulation result display section 371, an image height input field 372A, an image height setting slider 372B, a focus position input field 372C, and a focus position setting slider 372D. The simple lens emulation result display section 371 is a GUI similar to the simple lens emulation result display section 201. The image height input field 372A is a GUI similar to the image height input field 202A. The image height setting slider 372B is a GUI similar to the image height setting slider 202B. The focus position input field 372C is a GUI similar to the focus position input field 202C. The focus position setting slider 372D is a GUI similar to the focus position setting slider 202D.

[0127] Also, the lens setting section 322 has an aperture input field 373 as a GUI for setting the aperture. The aperture input field 373 is a GUI similar to the aperture input field 203A. Further, the lens setting section 322 has a focus position input field 374A and a focus position slider 374B for setting the focus position at the position of the knob as a GUI for setting the focus position. The focus position input field 374A is a GUI similar to the focus position input field 204A, and the focus position slider 374B is a GUI similar to the focus position slider 204B.

[0128] In addition, the lens setting unit 422 has a GUI for performing settings related to aberrations. For example, the lens setting unit 322 has, as such a GUI, a GUI for setting astigmatism, a GUI for setting spherical aberration, a GUI for setting coma aberration, a GUI for setting distortion aberration, and a GUI for setting field curvature.

[0129] For example, the lens setting unit 322 has an astigmatism coefficient input field 375A and an astigmatism coefficient slider bar 375B as a GUI for setting astigmatism. The astigmatism coefficient input field 375A is a GUI similar to the astigmatism coefficient input field 205A, and the astigmatism coefficient slider bar 375B is a GUI similar to the astigmatism coefficient slider bar 205B.

[0130] Also for example, the lens setting unit 322 has a spherical aberration coefficient input field 376A and a spherical aberration coefficient slider bar 376B as a GUI for setting spherical aberration. The spherical aberration coefficient input field 376A is a GUI similar to the spherical aberration coefficient input field 206A. The spherical aberration coefficient slider bar 376B is a GUI similar to the spherical aberration coefficient slider bar 206B.

[0131] Furthermore, for example, the lens setting unit 322 has a coma aberration coefficient input field 377A and a coma aberration coefficient slider bar 377B as a GUI for setting coma aberration. The coma aberration coefficient input field 377A is a GUI similar to the coma aberration coefficient input field 207A. The coma aberration coefficient slider bar 377B is a GUI similar to the coma aberration coefficient slider bar 207B.

[0132] Also for example, the lens setting unit 322 has a distortion aberration coefficient input field 378A and a distortion aberration coefficient slider bar 378B as a GUI for setting distortion aberration. The distortion aberration coefficient input field 378A is a GUI similar to the distortion aberration coefficient input field 208A. The distortion aberration coefficient slider bar 378B is a GUI similar to the distortion aberration coefficient slider bar 208B.

[0133] Furthermore, for example, the lens setting unit 322 has an image plane curvature coefficient input field 379A and an image plane curvature coefficient slider bar 379B as a GUI for setting the image plane curvature. The image plane curvature coefficient input field 379A is a GUI similar to the image plane curvature coefficient input field 209A. The image plane curvature coefficient slider bar 209B is a GUI similar to the image plane curvature coefficient slider bar 209B.

[0134] Also, the lens setting unit 322 has a read button 380 and a write button 381. The read button 380 is a GUI button similar to the read button 210. The write button 381 is a GUI button similar to the write button 211.

[0135] <Other image composition operation> As described above, a user or the like can perform an operation (also referred to as an other image composition operation) of synthesizing a synthesis target image with the captured image displayed on the composite image display unit 310. This other image composition operation is preferably performed, for example, after depth processing of the captured image.

[0136] For example, a user or the like determines the x and y positions of a synthesis target image (e.g., a CG (Computer Graphics) image) to be synthesized by a drag-and-drop operation. During dragging, the entire synthesis target image is displayed, and when dropped, occlusion with the captured image is reflected and displayed. For example, as shown in A of FIG. 18, when the synthesis target image 310A is located on the front side, the synthesis target image 310A is displayed with priority over the captured image (the portion of the captured image that overlaps with the synthesis target image 310A is displayed as being hidden by the synthesis target image 310A). Also, as shown in B of FIG. 18, when the synthesis target image 310A is located on the back side, the captured image is displayed with priority over the synthesis target image 310A (the portion of the synthesis target image 310A that overlaps with the captured image is displayed as being hidden by the captured image).

[0137] For example, the user or the like determines the depth position of the image to be synthesized by operating the mouse wheel while selecting the image to be synthesized. The size of the image to be synthesized may be controlled according to the depth position. Also, the image to be synthesized may be any image, such as a CG image or a captured image. Further, the number of images to be synthesized into the captured image is arbitrary. Furthermore, lens emulation may be performed on the synthesized space.

[0138] <2. First Embodiment> <Computer> Next, a device that executes the above application program will be described. FIG. 19 is a diagram showing a main configuration example of a computer which is an embodiment of an image processing apparatus to which the present technology is applied. The computer 500 shown in FIG. 19 is a device that installs and executes an application program (software) that performs lens emulation and other image synthesis processing as described above using a multi-viewpoint image. Here, the computer includes a computer incorporated in dedicated hardware, and a general-purpose personal computer or the like that can execute various functions by installing various programs. By executing the application program, the computer 500, for example, receives the selection of the lens to be emulated using the GUI (development screen 100) described above in <1. Lens Emulation>, generates an aberration table which is a table of converging light vectors due to the aberration of the selected lens, and performs a converging process to generate a captured image of the subject captured through the selected lens using the aberration table. By doing so, since the user's setting work for parameters related to aberration can be done to the extent of the lens selection work, lens emulation can be performed more easily.

[0139] Note that FIG. 19 shows the main components such as the processing unit and the data flow, but not all of them. That is, in the computer 500, there may be a processing unit not shown as a block in FIG. 19, or a process or data flow not shown as an arrow or the like in FIG. 19.

[0140] In the computer 500 shown in FIG. 19, a CPU (Central Processing Unit) 501, a ROM (Read Only Memory) 502, and a RAM (Random Access Memory) 503 are interconnected via a bus 504.

[0141] An input / output interface 510 is also connected to the bus 504. An input unit 511, an output unit 512, a storage unit 513, a communication unit 514, and a drive 515 are connected to the input / output interface 510.

[0142] The input unit 511 includes, for example, a keyboard, a mouse, a microphone, a touch panel, an input terminal, etc. The output unit 512 includes, for example, a display, a speaker, an output terminal, etc. The storage unit 513 includes, for example, a hard disk, a RAM disk, a non-volatile memory, etc. The communication unit 514 includes, for example, a network interface. The drive 515 drives a removable recording medium 521 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

[0143] In the computer 500 configured as described above, the CPU 501 loads and executes, for example, an application program stored in the storage unit 513 via the input / output interface 510 and the bus 504 into the RAM 503, thereby performing the above-described series of processes. The RAM 503 also appropriately stores data and the like necessary for the CPU 501 to execute various processes.

[0144] The application program executed by the computer 500 can be recorded on a removable recording medium 521 such as a package medium and applied. In that case, the application program can be installed in the storage unit 513 via the input / output interface 510 by mounting the removable recording medium 521 on the drive 515.

[0145] Also, this application program can be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting. In that case, this application program can be received by the communication unit 514 and installed in the storage unit 513.

[0146] In addition, this application program can be installed in advance in the ROM 502 and the storage unit 513.

[0147] FIG. 20 is a functional block diagram showing, as functional blocks, the functions realized by executing this application program in this computer 500. As shown in FIG. 20, the computer 500 can have functional blocks such as an emulation processing unit 531, a synthesis processing unit 532, a GUI processing unit 533, an image generation unit 534, and an output file generation unit 535 by executing the application program. Each functional block will be described below.

[0148] <Emulation Processing Unit> The emulation processing unit 531 performs processing related to the emulation of a lens using a development screen 100 as shown in FIG. 7, for example. For example, the emulation processing unit 531 can have functional blocks such as an input / output setting unit 541, a depth estimation unit 542, a ray interpolation unit 543, an aberration table generation unit 544, and a condensing processing unit 545.

[0149] The input / output setting unit 541 performs processes related to input / output settings using a GUI such as the input / output setting unit 111 or the input / output setting unit 311. For example, when a user or the like operates the input unit 511 based on the GUI of the input / output setting unit 111 or the input / output setting unit 311, the input / output setting unit 541 performs a process based on the operation received at the input unit 511. For example, when a file input instruction or an input file designation operation by a user or the like is received at the input unit 511, the input / output setting unit 541 can perform processes such as reading information on the designated file. Also, for example, when a file output instruction or an output folder designation operation by a user or the like is received at the input unit 511, the input / output setting unit 541 can generate a file in the designated output folder. The input / output setting unit 541 can perform this input / output setting, for example, before depth estimation.

[0150] The depth estimation unit 542 performs processes related to depth estimation using a GUI such as the depth estimation unit 112. For example, when a user or the like operates the input unit 511 based on the GUI of the depth estimation unit 112 or the like, the depth estimation unit 542 performs a process based on the operation received at the input unit 511. For example, when a parameter designation operation by a user or the like is received at the input unit 511, the depth estimation unit 542 can estimate the depth of a subject from a multi-viewpoint image using the designated parameters and generate depth data (which may be a depth map). The depth estimation unit 542 can perform this depth estimation, for example, before ray interpolation.

[0151] The ray interpolation unit 543 performs ray interpolation using multi-viewpoint images and depth data. Ray interpolation generates viewpoint images between the viewpoint images included in the captured images by interpolation processing in order to increase the number of viewpoint images used when generating virtual captured images. By increasing the number of viewpoint images in this way, a virtual image is generated from more viewpoint images including the viewpoint images generated by the interpolation processing. As a result, it becomes possible to generate a higher-definition virtual captured image. However, when a higher-definition virtual captured image is not generated, the processing of the ray interpolation unit 543 becomes unnecessary. The ray interpolation unit 543 can perform this ray interpolation, for example, before generating the aberration table.

[0152] The aberration table generation unit 544 performs processing related to the generation of an aberration table using, for example, GUIs such as the lens selection unit 121 and the lens setting unit 122 of the development screen 100. For example, when a user or the like operates the input unit 511 based on GUIs such as the lens selection unit 121 and the lens setting unit 122, the aberration table generation unit 544 performs processing based on the operation received in the input unit 511. For example, when a user or the like selects a lens to be emulated based on the lens selection unit 121 or customizes lens parameters based on a GUI such as the lens setting unit 122 and the operation is received in the input unit 511, the aberration table generation unit 544 uses the parameters of the selected lens or the customized lens parameters to generate an aberration table, which is a table of converging light ray vectors due to aberration.

[0153] The converging processing unit 545 performs converging processing using the aberration table and the multi-viewpoint images, and can generate a captured image (virtual captured image) of a subject imaged through the selected lens. Note that the converging processing unit 545 can perform this converging processing using multi-viewpoint images of an arbitrary resolution. For example, the converging processing unit 545 can perform converging processing using a multi-viewpoint image with a lower resolution than the original multi-viewpoint image obtained in actual imaging, and generate a virtual captured image with a lower resolution than the original multi-viewpoint image.

[0154] Note that after the depth map is determined by depth estimation, the lens emulation unit 113 becomes effective, and after the selection of the lens is accepted, ray interpolation may be performed by the ray interpolation unit 543 so that the interpolated ray information is retained. By doing so, it is possible to clearly separate the ray interpolation process and the depth estimation process, which have a large load. Also, by performing ray interpolation at the timing when the lens is selected, when the lens parameters are changed thereafter, a virtual captured image can be obtained only by generating an aberration table and performing a condensing process. That is, a virtual captured image (condensing process result) can be obtained at a higher speed.

[0155] <Other image synthesis processing unit> The other image synthesis processing unit 532 performs processing related to other image synthesis processing using, for example, the development screen 100 (GUI for other image synthesis processing corresponding to the tab 101C) shown in FIG. 14. For example, when a user or the like operates the input unit 511 based on the GUI, the other image synthesis processing unit 532 synthesizes a synthesis target image with the virtual captured image generated by the condensing process according to the operation received in the input unit 511, thereby generating a synthesized image.

[0156] <GUI processing unit> The GUI processing unit 533 performs processing related to the GUI displayed on the output unit 512 (monitor or the like). For example, the GUI processing unit 533 can cause the image generation unit 534 to generate a display image such as a GUI. Also, when the input unit 511 is operated based on the displayed GUI, the GUI processing unit 533 can control the emulation processing unit 531, the other image synthesis processing unit 532, the image generation unit 534, etc., and execute the processing corresponding to the operation. Further, the GUI processing unit 533 can acquire the information input to the input unit 511 based on the displayed GUI and supply it to the emulation processing unit 531, the other image synthesis processing unit 532, the image generation unit 534, etc. That is, the GUI processing unit 533 can control the reception of input information and the display of images.

[0157] <Image generation unit> The image generation unit 534 performs processing related to the generation of a display image (e.g., the development screen 100). For example, the image generation unit 534 generates the display image controlled by the GUI processing unit 503. At that time, the image generation unit 534 can generate the display image using, for example, information supplied from the GUI processing unit 503. For example, the image generation unit 534 can reflect information such as the file name and path input to the input unit 511 on the development screen 100 (that is, generate the development screen 100 including the file name and path). Also, the image generation unit 534 can reflect, for example, the virtual captured image generated in the light condensing processing unit 545 and the composite image generated in the other image composite processing unit 532 on the development screen 100 (that is, generate the development screen 100 on which the virtual captured image and the composite image are displayed). Furthermore, the image generation unit 534 can supply the generated display image to the output unit 512 (such as the monitor) and display it.

[0158] <Output file generation unit> The output file generation unit 535 performs processes related to the generation of output files. For example, the output file generation unit 535 can generate an output file for outputting the virtual captured image generated by the light condensing processing unit 545 or the composite image generated by the other image composite processing unit 532. The output file generation unit 535 can supply the generated output file to the storage unit 513 for storage, transmit it to another device via the communication unit 514, or record it on the removable recording medium 521 via the drive 515. That is, the output file generation unit 535 can control the output (storage, transmission, recording, etc.) of the output file (virtual captured image, composite image, etc.). For example, the output file generation unit 535 can cause the virtual captured image or the composite image to be output (stored, transmitted, recorded, etc.) at a resolution higher than the resolution during emulation (for example, a resolution equivalent to the original multi-viewpoint image). Of course, the output file generation unit 535 can also cause the virtual captured image or the composite image to be output (stored, transmitted, recorded, etc.) at the resolution during emulation. That is, the output file generation unit 535 can control the resolution of the virtual captured image or the composite image at the time of output.

[0159] By executing the application program to realize the functions indicated by such functional blocks, the computer 500 can perform the processes as described above in, for example, <1. Lens Emulation>, and can obtain the effects as described above.

[0160] <Flow of Lens Emulation Processing> Next, with reference to the flowcharts of FIGS. 21 and 22, an example of the flow of lens emulation processing realized by executing an application program in such a computer 500 will be described.

[0161] When the lens emulation processing is started, the GUI processing unit 533 causes the GUI (development screen 100) to be displayed on the monitor and starts accepting inputs in step S501 of FIG. 21.

[0162] In step S502, the input / output setting unit 541 performs input / output setting based on information input to the input unit 511, for example, based on the input / output setting unit 111 of the development screen 100 and the like.

[0163] In step S503, the depth estimation unit 542 performs depth estimation based on the result of the input / output setting performed in step S502 and information input to the input unit 511, for example, based on the depth estimation unit 112 of the development screen 100 and the like, and generates depth data.

[0164] In step S504, the GUI processing unit 533 determines, for example, whether the lens icon 181 is selected in the lens selection unit 121 of the development screen 100. If it is determined that the lens icon 181 is selected, the process proceeds to step S505.

[0165] In step S505, the ray interpolation unit 543 performs ray interpolation using the result of the input / output setting performed in step S502, the depth data generated in step S503, the parameters of the lens corresponding to the lens icon 181 selected in step S504, and the like. When the process of step S505 ends, the process proceeds to step S507.

[0166] Also, in step S504, if it is determined that the lens icon 181 is not selected, the process proceeds to step S506. In step S506, the GUI processing unit 533 determines, for example, whether the parameters are changed in the lens setting unit 122 of the development screen 100. If it is determined that the parameters are changed, the process proceeds to step S507.

[0167] In step S507, the aberration table generation unit 544 generates an aberration table using the set parameters (parameters of the selected lens or customized parameters).

[0168] In step S508, the condensing processing unit 545 performs condensing processing using the aberration table generated in step S507, and generates a virtual captured image captured through the selected lens.

[0169] In step S509, the image generation unit 534 generates a display image using the virtual captured image. In step S510, the image generation unit 534 causes the display image to be displayed on the monitor of the output unit 512. When the process of step S510 ends, the process proceeds to FIG. 22.

[0170] Also, in step S506, if it is determined that the parameters have not been changed, the process proceeds to FIG. 22.

[0171] In step S521 of FIG. 22, the output file generation unit 535 determines whether to output data. If it is determined to output, the process proceeds to step S522. In step S522, the output file generation unit 535 generates output data. In step S523, the output file generation unit 535 outputs the output data. For example, the output file generation unit 535 can store the generated output data in the storage unit 513, transmit it to another device via the communication unit 514, or record it on the removable recording medium 521 via the drive 515. When the process of step S523 ends, the process proceeds to step S524. Also, in step S521, if it is determined not to output data, the process proceeds to step S524.

[0172] In step S524, the GUI processing unit 533 determines whether to end this lens emulation process. If it is determined not to end, the process returns to step S502 in FIG. 21, and the subsequent processes are executed. Also, in step S524 of FIG. 22, if it is determined to end the lens emulation process, the lens emulation process ends.

[0173] By executing each process as described above, a user or the like can more easily perform lens emulation.

[0174] <Flow of Other Image Synthesis Process> Next, an example of the flow of an other image synthesis process realized by executing an application program on such a computer 500 will be described with reference to the flowchart of FIG. 23. When the other image synthesis process is started, each process from step S551 to step S554 is executed in the same manner as each process from step S501 to step S503 and step S505 of the lens emulation process (FIG. 20).

[0175] In step S555, the other image synthesis unit 532 sets the position of the image to be synthesized. In step S556, the other image synthesis unit 532 synthesizes the image to be synthesized with the virtual captured image to generate a synthesized image. In step S557, the image generation unit 534 generates a display image using the synthesized image and displays it on the monitor of the output unit 512.

[0176] Furthermore, for the synthesized image, each process from step S558 to step S561 is executed in the same manner as each process from step S521 to step S524 of FIG. 22. When it is determined in step S561 that the other image synthesis process is to be terminated, the other image synthesis process is terminated.

[0177] By executing each process as described above, a user or the like can more easily perform image synthesis.

[0178] <3. Supplementary Note> <Hardware> The above-described series of processes can be executed by software (application program) or by hardware.

[0179] <Application Target of the Present Technology> This technology can be applied to any configuration. For example, this technology can be implemented as a part of a device such as a processor as a system LSI (Large Scale Integration), a module using a plurality of processors, etc., a unit using a plurality of modules, etc., or a set with additional other functions added to the unit.

[0180] Also, for example, this technology can also be applied to a network system composed of a plurality of devices. For example, this technology may be implemented as cloud computing that is processed in a shared and collaborative manner by a plurality of devices via a network. For example, this technology may be implemented in a cloud service that provides services to any terminal such as a computer, a portable information processing terminal, an IoT (Internet of Things) device, etc.

[0181] Note that in this specification, a system means a collection of a plurality of components (devices, modules (parts), etc.), regardless of whether all the components are in the same housing. Therefore, a plurality of devices housed in separate enclosures and connected via a network, and a single device in which a plurality of modules are housed in one enclosure are both systems.

[0182] <Fields and Applications Applicable to this Technology> Systems, devices, processing units, etc. to which this technology is applied can be used in any field such as transportation, medical care, security, agriculture, livestock industry, mining, beauty, factories, home appliances, meteorology, natural monitoring, etc. Also, their applications are arbitrary.

[0183] <Others> The embodiments of this technology are not limited to the above-described embodiments, and various changes can be made without departing from the gist of this technology.

[0184] For example, the configuration described as one device (or processing unit) may be divided and configured as a plurality of devices (or processing units). Conversely, the configurations described as a plurality of devices (or processing units) above may be combined and configured as one device (or processing unit). Of course, it is also possible to add configurations other than those described above to the configuration of each device (or each processing unit). Furthermore, if the overall configuration and operation of the system are substantially the same, a part of the configuration of a certain device (or processing unit) may be included in the configuration of another device (or another processing unit).

[0185] Also, for example, the program described above may be executed on any device. In that case, it is only necessary that the device has the necessary functions (function blocks, etc.) and can obtain the necessary information.

[0186] Also, for example, each step of a flowchart may be executed by one device, or may be executed by a plurality of devices in cooperation. Furthermore, when a plurality of processes are included in one step, the plurality of processes may be executed by one device, or may be executed by a plurality of devices in cooperation. In other words, the plurality of processes included in one step can also be executed as the processes of a plurality of steps. Conversely, the processes described as a plurality of steps can also be executed as one step in a combined manner.

[0187] Also, for example, the program executed by a computer may be such that the processing of the steps of writing the program is executed in time series along the order described in this specification, or may be executed in parallel, or individually executed at a necessary timing such as when a call is made. That is, as long as there is no contradiction, the processing of each step may be executed in an order different from the order described above. Furthermore, the processing of the steps of writing this program may be executed in parallel with the processing of another program, or may be executed in combination with the processing of another program.

[0188] In addition, for example, multiple technologies related to the present technology can be independently implemented individually as long as there is no conflict. Of course, any multiple of the present technologies can also be implemented in combination. For example, part or all of the present technology described in any one of the embodiments can be implemented in combination with part or all of the present technology described in other embodiments. Also, part or all of any of the above-described present technologies can be implemented in combination with other technologies not described above.

[0189] Note that the present technology can also adopt the following configurations. (1) An aberration table generation unit that generates an aberration table, which is a table of converging light vectors due to the aberration of a lens to be emulated selected via a user interface, and a converging process unit that performs a converging process to generate a virtual captured image of a subject captured through the selected lens using the aberration table and the multi-viewpoint image An image processing apparatus comprising: (2) Further comprising a display control unit that causes the user interface to be displayed on a display unit, wherein the user interface includes icons corresponding to each lens that is a candidate for the emulation target, and the aberration table generation unit generates the aberration table using lens parameters corresponding to the icon selected via the user interface The image processing apparatus according to (1). (3) The user interface includes, as the icon, an icon corresponding to a lens with preset lens parameters (2) The image processing apparatus according to (2). (4) The user interface includes, as the icon, an icon corresponding to a lens whose lens parameters are set by the user (2) The image processing apparatus according to (2). (5) The user interface further includes the virtual captured image generated by the converging process (2) The image processing apparatus according to (2). (6) The light condensing processing unit generates the virtual captured image in which the drop position of the icon is focused by performing the light condensing processing in response to a drag-and-drop operation of the selected icon performed in the user interface. The image processing apparatus according to (5). (7) The light condensing processing unit generates the virtual captured image having a lower resolution than the multi-viewpoint image before being downsampled by performing the light condensing processing using the downsampled multi-viewpoint image. The user interface further includes the virtual captured image having a lower resolution than the multi-viewpoint image before being downsampled, which is generated by the light condensing processing. The image processing apparatus according to (5). (8) The apparatus further includes a storage control unit that stores the virtual captured image in a storage medium in a state having a higher resolution than the resolution when the virtual captured image is displayed as the user interface. The image processing apparatus according to (7). (9) The user interface includes a user interface for updating lens parameters corresponding to the selected icon. The aberration table generation unit generates the aberration table using the lens parameters updated via the user interface. The image processing apparatus according to (2). (10) The user interface includes the virtual captured image using a point light source as the subject, which represents the result of the light condensing processing. The image processing apparatus according to (2). (11) The user interface includes at least one of a user interface for setting the aperture and a user interface for setting the focus position. The image processing apparatus according to (2). (12) The user interface includes a user interface for setting aberration. The image processing apparatus according to (2). (13) Further comprising a ray interpolation unit that performs ray interpolation after selecting the lens to be emulated The image processing apparatus according to (1). (14) Further comprising a depth estimation unit that performs depth estimation before the ray interpolation is performed The image processing apparatus according to (13). (15) The user interface includes a user interface for setting the depth estimation, The depth estimation unit performs the depth estimation based on the setting related to the depth estimation set via the user interface. The image processing apparatus according to (14). (16) Further comprising an input / output setting unit that performs input / output setting before the depth estimation is performed The image processing apparatus according to (14). (17) The user interface includes a user interface for inputting information related to the input / output setting, The input / output setting unit performs the input / output setting based on the information related to the input / output setting input via the user interface. The image processing apparatus according to (16). (18) Further comprising another image synthesis processing unit that synthesizes the virtual captured image and the image to be synthesized The image processing apparatus according to (1). (19) Generating an aberration table that is a table of converging ray vectors due to the aberration of the lens to be emulated selected via the user interface, Performing a converging process of generating a virtual captured image of an object imaged through the selected lens using the aberration table and the multi-viewpoint image. Image processing method. (20) A computer, An aberration table generation unit that generates an aberration table that is a table of converging ray vectors due to the aberration of the lens to be emulated selected via the user interface, A condensing processing unit that performs a condensing process of generating a virtual captured image obtained by capturing a subject through the selected lens using the aberration table and the multi-viewpoint image A program that functions as

Explanation of Signs

[0190] 10 cameras, 100 developing screens, 101 tab display section, 102 GUI display section, 110 captured image display section, 111 input / output setting section, 112 depth estimation section, 113 lens emulation section, 121 lens selection section, 122 lens setting section, 171 lens icon display section, 172 add button, 173 delete button, 174 processing result storage section, 181 lens icon, 310 composite image display section, 311 input / output setting section, 313 lens emulation section, 321 lens selection section, 322 lens setting section, 500 computer, 531 emulation processing section, 532 other image composite processing section, 533 GUI processing section, 534 image generation section, 535 output file generation section, 541 input setting section, 542 depth estimation section, 543 ray interpolation section, 544 aberration table generation section, 545 condensing processing section

Claims

1. A display control unit that causes a display unit to display a user interface including icons corresponding to each lens that is a candidate for emulation target; An aberration table generation unit that generates an aberration table that is a table of converging light vectors due to aberrations of a lens to be emulated corresponding to the icon selected via the user interface, using lens parameters corresponding to the selected icon; A converging process unit that performs a converging process to generate a virtual captured image of a subject captured through the selected lens, using the aberration table and the multi-viewpoint image; comprising: The user interface further includes the virtual captured image generated by the converging process; The converging process unit performs the converging process in response to a drag-and-drop operation of the selected icon performed on the user interface, thereby generating the virtual captured image in which the drop position of the icon is in focus. An image processing apparatus.

2. The user interface includes, as the icon, an icon corresponding to a lens with preset lens parameters. The image processing apparatus according to claim 1.

3. The user interface includes, as the icon, an icon corresponding to a lens whose lens parameters are set by a user. The image processing apparatus according to claim 1.

4. The converging process unit performs the converging process using the downsampled multi-viewpoint image, thereby generating the virtual captured image with a lower resolution than the multi-viewpoint image before downsampling. The user interface further includes the virtual captured image with a lower resolution than the multi-viewpoint image before downsampling, generated by the converging process. The image processing apparatus according to claim 1.

5. Further comprising a storage control unit for storing the virtual captured image in a storage medium in a state with a higher resolution than the resolution when the virtual captured image is displayed as the user interface. The image processing apparatus according to claim 4.

6. The user interface includes a user interface for updating lens parameters corresponding to the selected icon. The aberration table generation unit generates the aberration table using the lens parameters updated via the user interface. The image processing apparatus according to claim 1.

7. The user interface includes a virtual captured image having a point light source as the subject, which represents the result of the condensing process. The image processing apparatus according to claim 1.

8. The user interface includes at least one of a user interface for setting aperture and a user interface for setting focus position. The image processing apparatus according to claim 1.

9. The user interface includes a user interface for setting aberration. The image processing apparatus according to claim 1.

10. Further comprising a ray interpolation unit for performing ray interpolation after selecting the lens to be emulated. The image processing apparatus according to claim 1.

11. Further comprising a depth estimation unit for performing depth estimation before the ray interpolation is performed. The image processing apparatus according to claim 10.

12. The user interface includes a user interface for setting the depth estimation. The depth estimation unit performs the depth estimation based on the setting related to the depth estimation set via the user interface. The image processing apparatus according to claim 11.

13. The apparatus further includes an input / output setting unit that performs input / output setting before the depth estimation is performed. The image processing apparatus according to claim 11.

14. The user interface includes a user interface for inputting information related to the input / output setting, and the input / output setting unit performs the input / output setting based on the information related to the input / output setting input via the user interface. The image processing apparatus according to claim 13.

15. The apparatus further includes another image synthesis processing unit that synthesizes the virtual captured image and the synthesis target image. The image processing apparatus according to claim 1.

16. Cause a display unit to display a user interface including icons corresponding to respective lenses that are candidates for an emulation target, generate an aberration table that is a table of converging light vectors due to aberrations of the lens that is the emulation target corresponding to the icon selected via the user interface, using lens parameters corresponding to the selected icon, perform a focusing process of generating a virtual captured image of a subject imaged through the selected lens using the aberration table and the multi-viewpoint image in response to a drag-and-drop operation of the selected icon performed in the user interface, thereby generating a virtual captured image in which the drop position of the icon is in focus, and including the generated virtual captured image in the user interface An image processing method.

17. A computer, A display control unit that causes a display unit to display a user interface including icons corresponding to each lens that is a candidate for emulation; An aberration table generation unit that generates an aberration table that is a table of converging light vectors due to aberrations of a lens to be emulated corresponding to the icon selected via the user interface, using lens parameters corresponding to the selected icon; A converging processing unit that performs a converging process of generating a virtual captured image obtained by capturing a subject through the selected lens, using the aberration table and the multi-viewpoint image; functioning; The user interface further includes the virtual captured image generated by the converging process; The converging processing unit generates the virtual captured image in which the drop position of the icon is focused by performing the converging process in response to a drag-and-drop operation of the selected icon performed in the user interface; Program.

Images