Image output device and image output method

Abstract

[Problem] To improve the depth resolution of a three-dimensional image in light field imaging. [Solution] An image output device (10A) is provided with a three-dimensional image reconstruction unit (100A) for outputting a composite three-dimensional image by using: three-dimensional images respectively output by two light field imaging systems (200-1, 200-2) in which lens groups (210-1, 210-2) having a main lens and a lens array are arranged in a state in which optical axes (250-1, 250-2) intersect with each other; and an angle (θ) at which the optical axes intersect with each other.

Description

Image output device and image output method

[0001] The disclosed technology relates to an image output technology for outputting a three-dimensional image.

[0002] Three-dimensional image capture technology is based on the technique of restoring depth information from two or more images based on parallax information. One image output technology that uses this technique to output three-dimensional images is called light field imaging, which combines a main lens and a lens array to construct a three-dimensional image using images captured by a single image sensor. For example, Patent Literature 1 describes light field imaging, in which a single image sensor captures multiple images via a main lens and a lenslet array (equivalent to the above-mentioned "lens array"). In light field imaging, the positions of multiple refocus planes are calculated using the distance from the main lens to the focal plane, and the spacing between the refocus planes increases exponentially with increasing distance from the focal plane of the main lens. These refocus planes define the depth length of a voxel, the smallest unit of three-dimensional space used to construct a three-dimensional image.

[0003] Special Publication No. 2016-511980

[0004] Conventional light field imaging had the problem that the depth resolution of three-dimensional images was significantly degraded because the depth length of voxels increased exponentially as they moved away from the focal plane of the main lens.

[0005] The present disclosure is intended to solve the above-mentioned problems, and aims to improve the depth resolution of three-dimensional images in light field imaging.

[0006] The image output device of the present disclosure is an image output device that outputs a three-dimensional image using images captured through a lens group having a main lens and a lens array, and is equipped with a three-dimensional image reconstruction unit that outputs a composite three-dimensional image using each image captured through two sets of lens groups whose optical axes intersect with each other, and the angle formed by the optical axes of the two sets of lens groups.

[0007] According to the present disclosure, it is possible to achieve an effect of improving the depth resolution of a three-dimensional image in light field imaging.

[0008] FIG. 1 is a diagram illustrating an example of a basic configuration of an image output device according to a first embodiment of the present disclosure. FIG. 2 is a diagram illustrating an image of a three-dimensional space configured of voxels. FIG. 3 is a diagram illustrating an image of a refocus plane obtained by a refocus process. FIG. 4 is a diagram illustrating a three-dimensional space configured of voxels viewed from above. FIG. 5 is a diagram illustrating an image of a three-dimensional image obtained by a light field imaging system that combines a lens group, an imaging unit, and a three-dimensional image constructing unit. FIG. 6 is a diagram illustrating an image of a three-dimensional image obtained by a conventional light field imaging system. FIG. 7 is a diagram illustrating an image of a combination process according to the first embodiment of the present disclosure. FIG. 8 is a flowchart illustrating an example of processing by the image output device according to the first embodiment of the present disclosure. FIG. 9 is a diagram illustrating an example of a configuration when the image output device according to the first embodiment of the present disclosure is applied to an image output system. FIG. 10 is a diagram illustrating an image of a light field imaging system that outputs a three-dimensional image used in the image output device according to the first embodiment of the present disclosure. FIG. 11 is a diagram illustrating an example of implementation of the image output device and the image output system according to the first embodiment of the present disclosure. FIG. 12 is a flowchart illustrating an example of processing by the image output device or the image output system according to the first embodiment of the present disclosure. FIG. 13 is a flowchart showing an example of combination processing in processing by the image output device according to the first embodiment of the present disclosure. FIG. 14 is a diagram showing an image of a voxel after processing by the image output device according to the second embodiment of the present disclosure. FIG. 15 is a diagram showing an image of a voxel after processing by the image output device according to the third embodiment of the present disclosure. FIG. 16 is a diagram showing an image of a voxel after processing by the image output device according to the fourth embodiment of the present disclosure. FIG. 17 is a diagram showing an example configuration of an image output device according to the fifth embodiment of the present disclosure and an image output system to which the image output device is applied. FIG. 18 is a diagram showing an example implementation of the image output device and image output system according to the fifth embodiment of the present disclosure. FIG. 19 is a flowchart showing an example of processing by the image output device or the image output system according to the fifth embodiment of the present disclosure.Fig. 20 is a diagram illustrating a configuration example of an image output device according to a sixth embodiment of the present disclosure and an image output system to which the image output device is applied. Fig. 21 is a diagram illustrating an image of an implementation example of the image output device and image output system according to the sixth embodiment of the present disclosure. Fig. 22 is a flowchart illustrating an example of processing by the image output device or image output system according to the sixth embodiment of the present disclosure. Fig. 23 is a flowchart illustrating an example of processing order command processing by the image output device or image output system according to the sixth embodiment of the present disclosure. Fig. 24 is a diagram illustrating a first example of a hardware configuration for realizing functions according to the configuration of the present disclosure. Fig. 25 is a diagram illustrating a second example of a hardware configuration for realizing functions according to the configuration of the present disclosure.

[0009] To explain the present disclosure in more detail, embodiments of the present disclosure will be described below with reference to the accompanying drawings. The embodiments of the present disclosure are intended to output a three-dimensional image using light field imaging technology. The embodiments of the present disclosure are based on the idea of ​​capturing images by overlapping two or more imaging regions using a light field imaging system (LFIS) configuration equipped with two or more main lenses having different focal planes and lens arrays having different F-numbers. The embodiments of the present disclosure are based on the idea that a three-dimensional image is synthesized from two light field images, and the lateral pixels compensate for the depth pixel size of the overlapping region, thereby improving depth resolution because the lateral pixels correspond to the depth pixels.

[0010] First Embodiment In the first embodiment, a basic form of the present disclosure will be described.

[0011] An example configuration of an image output device according to a first embodiment of the present disclosure will be described. Fig. 1 is a diagram showing an example basic configuration of an image output device according to the first embodiment of the present disclosure. The image output device 10 outputs a three-dimensional image using an image captured through a lens group having a main lens and a lens array. The image output device 10 shown in Fig. 1 is configured to include a three-dimensional image reconstruction unit 100.

[0012] The three-dimensional image reconstruction unit 100 outputs a composite three-dimensional image using each image captured through the two lens groups whose optical axes intersect with each other, and the angle formed by the optical axes of the two lens groups. The three-dimensional image reconstruction unit 100 shown in FIG. 1 includes a three-dimensional image acquisition unit 110 (110-1, 110-2, ..., 110-n (2≦n)), an angle acquisition unit 120 (120-1, 120-2, ..., 120-n (2≦n)), and a combination processing unit 130.

[0013] The three-dimensional image acquisition units 110 (110-1, 110-2, ..., 110-n (2≦n)) acquire a three-dimensional image constructed using images captured through each lens group (for example, for each of two lens groups formed by a combination of lens group 200-1 and lens group 200-2, which will be described later). The three-dimensional image acquisition units 110 (110-1, 110-2, ..., 110-n (2≦n)) acquire, for example, three-dimensional images output from each of a plurality of light field imaging systems (see Figures 9, 10, and 11, which will be described later). The plurality of three-dimensional image acquisition units 110 (110-1, 110-2, ..., 110-n (2≦n)) can also be collectively referred to as a "three-dimensional image acquisition unit group."

[0014] The angle acquisition units 120 (120-1, 120-2, ..., 120-n (2≦n)) acquire the angle formed by the optical axes of each of two lens groups (for example, two lens groups formed by a combination of lens group 200-1 and lens group 200-2, which will be described later). The multiple angle acquisition units 120 (120-1, 120-2, ..., 120-n (2≦n)) can also be collectively referred to as an "angle acquisition unit group." For example, the angle acquisition unit 120-1 or the angle acquisition unit 120-2 acquires the angle formed by the optical axes of each of two lens groups (for example, two lens groups formed by a combination of lens group 200-1 and lens group 200-2, which will be described later). In addition, when the system is composed of only two lens groups (for example, when the two lens groups are composed of only a combination of lens group 200-1 and lens group 200-2, which will be described later), it is sufficient to use one lens group as a reference to obtain the angle of the optical axis of the other lens group, so the system may be configured to have only one of angle acquisition unit 120-1 or angle acquisition unit 120-2.

[0015] The combination processing unit 130 combines multiple three-dimensional images to output a three-dimensional image (a composite three-dimensional image). The combination processing unit 130 outputs a composite three-dimensional image, which is an image obtained by combining three-dimensional images, using the three-dimensional images acquired by the three-dimensional image acquisition units 110 (110-1, 110-2, ..., 110-n (2≦n)) and the angles acquired by the angle acquisition units 120 (120-1, 120-2, ..., 120-n (2≦n)). Specifically, when combining two three-dimensional images, the combination processing unit 130 uses the two three-dimensional images and the angle formed by the optical axes of two sets of lens groups (for example, two sets of lens groups formed by the combination of lens group 200-1 and lens group 200-2, described below) to calculate pixel values ​​for overlapping portions of voxels in one three-dimensional image and voxels in the other three-dimensional image, respectively. The combination processing unit 130 reconstructs a composite three-dimensional image using the calculated pixel values ​​for each voxel.

[0016] In addition to the above components, the image output device 10 also includes a control unit (not shown), a storage unit (not shown), and a communication unit (not shown). The control unit (not shown) controls the entire image output device 10 and each of its components. The control unit (not shown), for example, starts up the image output device 10 in response to external commands. The control unit (not shown) also controls the state of the image output device 10 (operating state, such as startup, shutdown, or sleep). The storage unit (not shown) stores various data used by the image output device 10. For example, the storage unit (not shown) stores output (output data) from each component of the image output device 10 and outputs data requested by each component to the requesting component. The communication unit (not shown) communicates with external devices or between internal components. For example, communication is performed between the image output device 10 (100A) and a peripheral device (e.g., a display device (not shown). For example, if the image output device 10 and the peripheral device are not connected via a wire, the communication unit (not shown) functions to communicate between the image output device 10 and the peripheral device. Furthermore, a communication unit (not shown) has a function of communicating between the image output device 10 and a server device (not shown). The server device may be configured to perform part of the functions of the image output device. The control unit (not shown), the storage unit (not shown), and the communication unit (not shown) each have the same functions in the embodiments described below.

[0017] Here, an image of processing by the image output device of the present disclosure will be described. FIG. 2 is a diagram showing an image of a three-dimensional space 1000 configured with voxels. FIG. 3 is a diagram showing an image of a refocus plane obtained by refocus processing. FIG. 4 is a diagram showing a three-dimensional space configured with voxels viewed from above. FIG. 5 is a diagram showing an image of a case where a three-dimensional image is obtained by a set of light field imaging systems that combines a lens group, an imaging unit, and a three-dimensional image construction unit. FIG. 6 is a diagram showing an image of a three-dimensional image obtained by a conventional light field imaging system. FIG. 7 is a diagram showing an image of combination processing according to the first embodiment of the present disclosure. The three-dimensional space 1000 is refocused along the optical axis (axis extending in the direction of arrow α) of the main lens in the lens group, with the position of the focal plane 1010 of the main lens as a reference, to define refocus planes (1020, 1021, 1022, 1023, 1024, 1025, ...), and the depth direction of the voxels in the three-dimensional space 1000 is divided by these refocus planes (1020, 1021, 1022, 1023, 1024, 1025, ...). Figure 4 shows a view as viewed in the direction indicated by arrow β in Figures 2 and 3. As shown in Figure 4, the voxels 1030 (1034, 1035), which are units of three-dimensional pixels, become longer in the depth direction (z direction) the further they are from the focal plane 1010 of the main lens. Here, consider a case where an image capture object 1100 exists between a refocus plane 1026 and a refocus plane 1027, as shown in FIG. 5 . It is assumed that the image capture object 1100 is actually a spherical object. Each of the three central voxels shown in FIG. 5 generates a pixel value representing the image capture object 1100. However, because the voxels are long in the depth direction between the refocus plane 1026 and the refocus plane 1027, a three-dimensional image corresponding to the length is constructed. In a conventional light field imaging system, a three-dimensional image 1110 such as that shown in FIG. 6 is output from the pixel values ​​of the voxels between the refocus plane 1026 and the refocus plane 1027 shown in FIG. 5 .In contrast, when a three-dimensional image is reconstructed using the concept of the present disclosure, the voxel units can be expressed as 3-II and 2-II, for example, and three-dimensional pixel values ​​can be expressed using smaller voxels than the conventional single voxel, as shown in Fig. 7. By using three-dimensional pixel values ​​in voxel units, it is possible to achieve a representation closer to the real thing through higher resolution, and it is possible to increase the resolution of three-dimensional images.

[0018] A process according to the first embodiment of the present disclosure will be described below. Fig. 8 is a flowchart showing an example of a process performed by the image output device according to the first embodiment of the present disclosure.

[0019] The process shown in Figure 8 is an image output method using an image output device that outputs a three-dimensional image using images captured through a lens group having a main lens and a lens array, and includes a three-dimensional image reconstruction step. In the three-dimensional image reconstruction step, a three-dimensional image reconstruction unit of the image output device outputs a composite three-dimensional image using each image captured through two sets of lens groups whose optical axes intersect with each other, and the angle formed by the optical axes of the two sets of lens groups. The process shown in Figure 8 will be described in detail. The image output device 10 starts the process, for example, when a control unit (not shown) of the image output device 10 receives an image output command.

[0020] The image output device 10 then executes a three-dimensional image acquisition process (step ST110). In the three-dimensional image acquisition process, the three-dimensional image acquisition units 110 (110-1, 110-2, ..., 110-n (2≦n)) of the image output device 10 acquire, for example, three-dimensional images output from a plurality of light field imaging systems. The three-dimensional image acquisition units 110 (110-1, 110-2, ..., 110-n (2≦n)) acquire three-dimensional images constructed using images captured through each lens group (for example, for each of two lens groups formed by a combination of lens group 200-1 and lens group 200-2, which will be described later). The three-dimensional image acquisition units 110 (110-1, 110-2, ..., 110-n (2≦n)) each output the acquired three-dimensional images to the combination processing unit 130.

[0021] Next, the image output device 10 executes an angle acquisition process (step ST120). In the angle acquisition process, the angle acquisition units 120 (120-1, 120-2, ..., 120-n (2≦n)) of the image output device 10 acquire the angle formed by the optical axes of the two lens groups (for example, two lens groups formed by the combination of lens group 200-1 and lens group 200-2, which will be described later). The angle acquisition units 120 (120-1, 120-2, ..., 120-n (2≦n)) each output the acquired angle to the combination processing unit 130.

[0022] The image output device 10 then executes a combination process (step ST130). In the combination process, the combination processing unit 130 of the image output device 10 uses the 3D images acquired by the 3D image acquisition units 110 (110-1, 110-2, ..., 110-n (2≦n)) and the angles acquired by the angle acquisition units 120 (120-1, 120-2, ..., 120-n (2≦n)) to output a composite 3D image, which is an image combining the 3D images. Specifically, the combination processing unit 130 acquires, for example, the 3D image acquired by the 3D image acquisition unit 110-1, the 3D image acquired by the 3D image acquisition unit 110-2, the angles acquired by the angle acquisition unit 120-1, and the angles acquired by the angle acquisition unit 120-2. The combination processing unit 130 combines the three-dimensional image acquired by the three-dimensional image acquisition unit 110-1 and the three-dimensional image acquired by the three-dimensional image acquisition unit 110-2 from the angles acquired by the angle acquisition unit 120-1 and the angle acquired by the angle acquisition unit 120-2, and generates and outputs a composite three-dimensional image reconstructed from subdivided voxels.

[0023] Next, the image output device 10 executes a composite 3D image output process (step ST140). In the composite 3D image output process, the combination processing unit 130 of the image output device 10 outputs the generated composite 3D image to an external device. The combination processing unit 130 outputs the composite 3D image to, for example, an external display device.

[0024] After executing the composite 3D image output process (step ST140), the image output device 10 proceeds to an end determination process (step ST150). In the end determination process, a control unit (not shown) of the image output device 10 determines whether to end the process of the image output device 10. The control unit (not shown) determines whether to end the process of the image output device 10, for example, in accordance with an external end command or an execution program. If the control unit (not shown) determines not to end the process of the image output device 10 ("NO" in step ST150), the process proceeds to step ST110, and the process is repeated from step ST110. If the control unit (not shown) determines to end the process of the image output device 10 ("YES" in step ST150), the image output device 10 terminates the process.

[0025] Next, a configuration example of a system including an image output device according to the present embodiment will be described. FIG. 9 is a diagram illustrating a configuration example in which the image output device according to the first embodiment of the present disclosure is applied to an image output system. FIG. 10 is a diagram illustrating an image of a light field imaging system that outputs a three-dimensional image used in the image output device according to the first embodiment of the present disclosure. FIG. 11 is a diagram illustrating an image of an implementation example of the image output device and image output system according to the first embodiment of the present disclosure. The image output system 1A includes at least two or more sets of main lenses having different focal planes and lens arrays having different F-numbers matched to the main lenses. The image output system 1A also includes a three-dimensional image reconstruction unit 100A. The image output system 1A uses an LFIS (light field imaging system) equipped with two or more main lenses having different focal planes and lens arrays having different F-numbers to capture images with two or more overlapping imaging regions, thereby improving resolution in the depth direction. The image output system 1A shown in FIG. 9 includes an image output device 10A, an LFIS 200 (200-1, 200-2, . . . , 200-n (2≦n)), and an image capture synchronization unit 300.

[0026] 9 includes a plurality of LFIS (Light Field Imaging Systems) 200 (200-1, 200-2, ..., 200-n (2≦n)). Each of the LFIS 200 (200-1, 200-2, ..., 200-n (2≦n)) includes a combination of a lens group 210-N (N=1, 2, ..., n, 2≦n), an imaging unit 220-N (N=1, 2, ..., n, 2≦n), and a three-dimensional image constructing unit 230-N (N=1, 2, ..., n, 2≦n). That is, the image output system 1A is configured to include a plurality of lens groups 210 (210-1, 210-2, ..., 210-n (2≦n)), a plurality of imaging units 220 (220-1, 220-2, ..., 220-n (2≦n)), and a plurality of three-dimensional image construction units 230 (230-1, 230-2, ..., 230-n (2≦n)).

[0027] The lens group 210-N (N=1, 2, ..., n, 2≦n) is configured to have a main lens unit 211-N (N=1, 2, ..., n, 2≦n) and a lens array unit 212-N (N=1, 2, ..., n, 2≦n). The lens group 210-1 shown in FIG. 9 has a main lens unit 211-1 and a lens array unit 212-1. The lens group 210-2 shown in FIG. 9 has a main lens unit 211-2 and a lens array unit 212-2. The main lens unit 211-N (N=1, 2, ..., n, 2≦n) is configured by a lens (main lens). The lens array unit 212-N (N=1, 2, ..., n, 2≦n) is configured by a lens array. That is, the plurality of lens groups 210 (210-1, 210-2, ..., 210-n (2≦n)) each have a main lens unit 211 (211-1, 211-2, ..., 211-n (2≦n)) and a lens array unit 212 (212-1, 212-2, ..., 212-n (2≦n)). The distance between the lens array of the lens array unit 212-N (N=1, 2, ..., n, 2≦n) and the imaging element of the imaging unit 220-N (N=1, 2, ..., n, 2≦n) is configured to be the same as the focal length of the lens array.

[0028] The imaging unit 220-N (N=1, 2, ..., n, 2≦n) is configured to include an imaging element, and outputs imaging data captured by the imaging element via a lens group 210-N (N=1, 2, ..., n, 2≦n). That is, the multiple imaging units 220 (220-1, 220-2, ..., 220-n (2≦n)) capture images via the lens group 210 (210-1, 210-2, ..., 210-n (2≦n)) for each lens group 210 (210-1, 210-2, ..., 210-n (2≦n)) and output imaging data. The multiple imaging units 220 (220-1, 220-2, ..., 220-n (2≦n)) can also be collectively referred to as an "imaging unit group."

[0029] On the optical axis of the main lens in main lens unit 211-N (N=1, 2, ..., n, 2≦n), main lens unit 211-N (N=1, 2, ..., n, 2≦n), lens array unit 212-N (N=1, 2, ..., n, 2≦n), and imaging unit 220-N (N=1, 2, ..., n, 2≦n) are arranged in that order, and the image through the main lens is passed through the lens array to form the same number of images as the number of arrays, and these are captured by the imaging element of imaging unit 220-N (N=1, 2, ..., n, 2≦n).

[0030] The three-dimensional image construction units 230-N (N = 1, 2, ..., n, 2 ≦ n) construct a three-dimensional image using images captured by the corresponding imaging units 220-N (N = 1, 2, ..., n, 2 ≦ n). That is, the multiple three-dimensional image construction units 230 (230-1, 230-2, ..., 230-n (2 ≦ n)) construct a three-dimensional image using images captured by the imaging units 220 (220-1, 220-2, ..., 220-n (2 ≦ n)) for each imaging unit 220 (220-1, 220-2, ..., 220-n (2 ≦ n)). The multiple three-dimensional image construction units 230 (230-1, 230-2, ..., 230-n (2 ≦ n)) can also be collectively referred to as a "group of three-dimensional image construction units."

[0031] The imaging synchronization unit 300 synchronizes two sets of LFIS 200 (200-1, 200-n (2≦n)) out of the LFIS 200 (200-1, 200-2, ..., 200-n (2≦n)). The imaging synchronization unit 300 is composed of, for example, a TTL signal generator, and outputs a TTL signal. The TTL signal generator may be any device that can continuously output a rectangular wave at a specific timing. In the present disclosure, it is sufficient that imaging by the two sets of LFIS 200 (200-1, 200-n (2≦n)) is synchronized, and the imaging synchronization unit 300 is not necessarily provided.

[0032] The image output device 10A shown in FIG. 9 is configured similarly to the image output device 10 already described. Regarding the image output device 10A, the following description will mainly focus on the details not mentioned in the description of the image output device 10. The image output device 10A shown in FIG. 9 includes a three-dimensional image reconstruction unit 100A. The three-dimensional image reconstruction unit 100A includes a three-dimensional image acquisition unit 110A (110A-1, 110A-2, ..., 110A-n (2≦n)), an angle acquisition unit 120A (120A-1, 120A-2, ..., 120A-n (2≦n)), and a combination processing unit 130A. The three-dimensional image acquisition unit 110A (110A-1, 110A-2, ..., 110A-n (2≦n)) acquires a three-dimensional image constructed by a three-dimensional image construction unit 230 (230-1, 230-2, ..., 230-n (2≦n)). Specifically, the three-dimensional image acquisition unit 110A-1 acquires a three-dimensional image constructed by the three-dimensional image construction unit 230-1. The three-dimensional image acquisition unit 110A-2 acquires a three-dimensional image constructed by the three-dimensional image construction unit 230-2. The three-dimensional image acquisition unit 110A-n (2≦n) acquires a three-dimensional image constructed by the three-dimensional image construction unit 230-n (2≦n). Note that the image output device 10A may be configured to include the range shown in FIG. 9 and the range of the image output system 1A. For example, the image output device 10A may be configured to include all or part of the multiple LFISs 200 (200-1, 200-2, ..., 200-n (2≦n)).

[0033] The two sets of LFIS 200-1 and 200-2 are configured so that the focal lengths of the main lens units 211-1 and 211-2 are different from each other, as shown in Fig. 10. Here, the F-number of the lens array is expressed by the following equation (1): F=f ＬＡ / P (1) "P" indicates the pitch of the lens array. ＬＡ " indicates the focal length of the lens array. When two sets of LFIS 200 (200-1, 200-2) are configured so that the focal lengths of the main lens units 211 (211-1, 211-2) are different, the F-number F1 of the lens array unit 212-1 and the F-number F2 of the lens array unit 212-2 are different F-numbers, according to the following formula (1). However, the present disclosure is not limited to this, and as shown in the embodiment described later, the focal lengths of the main lens units 211 (211-1, 211-2) may be the same. The focal length may be a distance far from the main lens units 211 (211-1, 211-2) or a distance close to the main lens units 211 (211-1, 211-2).

[0034] 11 shows a configuration image of an image output system 1A or an image output device 10A in which two sets of LFIS 200-1 and 200-2 shown in FIG. 10 are arranged so that the optical axes 250-1 and 250-2 of the main lens units 211-1 and 211-2 intersect. If the angle θ of the optical axes, which is the angle at the position where the optical axes intersect, is defined as the angle D, which indicates the depth resolution, ＯＬ is expressed as the following equation (2), and it can be seen that the configuration of the present disclosure improves the resolution. ＯＬ = 2d Ｌａｔ (cos(θ / 2) / sinθ) ...(2) “d Ｌａｔ " indicates the distance from the position of the main lens to the position of the focal plane. Or, the value D shown as in the following equation (3) ＯＬ It can be seen that the resolution is improved. ＯＬ = 2P(1-f ＬＡ / zM ２) ... (3) "z" indicates the distance from the position of the main lens to the position of the focal plane. "M" indicates the magnification (magnification). In this way, high resolution can be achieved by using the XY direction data from the two sets of LFIS 200-1 and 200-2.

[0035] The image output system 1A or the image output device 10A may be configured to include a light source 400 that irradiates light toward an imaging range or an imaging target, as shown in Fig. 11. The light source 400 is, for example, a point light source. The light source 400 is also, for example, a light source that irradiates diffused light. The light source 400 is also, for example, a light source that emits collimated light.

[0036] An example of processing in the image output system 1A or the image output device 10A will be described. Fig. 12 is a flowchart showing an example of processing in the image output device or the image output system according to the first embodiment of the present disclosure. Fig. 13 is a flowchart showing an example of combination processing in the processing in the image output device according to the first embodiment of the present disclosure. The processing shown in Figs. 12 and 13 is an image output method by the image output system or the image output device.

[0037] The processing shown in Fig. 12 will be described in detail. In the following description, it is assumed that two sets of lens groups of LFIS 200-N (N = 1, 2, ..., n (2 < n)), which is a combination of lens groups 210-N (N = 1, 2, ..., n (2 < n)), imaging unit 220-N (N = 1, 2, ..., n (2 < n)), and three-dimensional image construction unit 230-N (N = 1, 2, ..., n (2 < n)), are lens groups 210-1 and 210-2, and that two sets of LFIS are LFIS 200-1 and 200-2. The image output system 1A or image output device 10A starts processing when, for example, a control unit (not shown) of image output device 10A receives an image output command.

[0038] The image output system 1A or the image output device 10A then executes a shooting synchronization process (step ST1110). In the shooting synchronization process, the shooting synchronization unit 300 outputs a TTL signal to the two sets of LFIS 200-1 and LFIS 200-2, which synchronizes shooting in the two sets of LFIS 200-1 and LFIS 200-2 among the combinations of the lens group 210-N (N=1, 2, ..., n (2≦n)), the imaging unit 220-N (N=1, 2, ..., n (2≦n)), and the three-dimensional image construction unit 230-N (N=1, 2, ..., n (2≦n)).

[0039] The image output system 1A or the image output device 10A then executes an imaging data acquisition process (step ST1120). In the imaging data acquisition process, the multiple imaging units 220 (220-1, 220-2) capture images via the lens group 210 (210-1, 210-2) for each lens group 210 (210-1, 210-2) and output the imaging data. Here, the imaging unit 220-1 captures images via the lens group 210-1 and outputs the imaging data to the three-dimensional image construction unit 230-1. Also, the imaging unit 220-2 captures images via the lens group 210-2 and outputs the imaging data to the three-dimensional image construction unit 230-2.

[0040] The image output system 1A or the image output device 10A then executes a three-dimensional image construction process (step ST1130). In the three-dimensional image construction process, the three-dimensional image construction unit 230 (230-1, 230-2) constructs a three-dimensional image for each of the imaging units 220 (220-1, 220-2) using the images captured by the imaging units 220 (220-1, 220-2). Here, the three-dimensional image construction unit 230-1 constructs a three-dimensional image using the images captured by the imaging unit 220-1 and outputs the three-dimensional image to the three-dimensional image acquisition unit 110A-1. Furthermore, the three-dimensional image construction unit 230-2 constructs a three-dimensional image using the images captured by the imaging unit 220-2 and outputs the three-dimensional image to the three-dimensional image acquisition unit 110A-2.

[0041] The image output device 10A then executes a three-dimensional image acquisition process (step ST1140). In the three-dimensional image acquisition process, the three-dimensional image acquisition units 110A (110A-1, 110A-2, ..., 110A-n (n≧2)) of the image output device 10A acquire, for example, three-dimensional images output from each of a plurality of light field imaging systems. The three-dimensional image acquisition units 110 (110A-1, 110A-2) acquire, for each of the lens groups 210 (210-1, 210-2), three-dimensional images constructed using images captured via the lens groups 210 (210-1, 210-2). The three-dimensional image acquisition units 110A (110A-1, 110A-2, ..., 110A-n (n≧2)) each output the acquired three-dimensional images to the combination processing unit 130A. Three-dimensional image acquisition unit 110A-1 acquires a three-dimensional image constructed using images captured via lens group 210-1. Three-dimensional image acquisition unit 110A-2 acquires a three-dimensional image constructed using images captured via lens group 210-2.

[0042] Next, the image output device 10A executes angle acquisition processing (step ST1150). In the angle acquisition processing, the angle acquisition units 120A (120A-1, 120A-2) of the image output device 10A acquire the angle formed by the optical axes of the two lens groups (for example, two lens groups formed by the combination of lens group 200-1 and lens group 200-2). The angle acquisition units 120A (120-1, 120-2) each output the acquired angle to the combination processing unit 130. Alternatively, the angle acquisition units 120A (120-1, 120-2) output the angle θ formed by the optical axis 250-1 of the main lens unit 211-1 and the optical axis 250-2 of the main lens unit 211-2 to the combination processing unit 130.

[0043] The image output device 10A then executes a combination process (step ST1160). In the combination process, the combination processing unit 130A of the image output device 10A uses the 3D images acquired by the 3D image acquisition units 110A (110A-1 and 110A-2) and the angles acquired by the angle acquisition units 120A (120A-1 and 120A-2) to output a composite 3D image, which is an image combining the 3D images. Specifically, the combination processing unit 130A acquires, for example, the 3D image acquired by the 3D image acquisition unit 110A-1, the 3D image acquired by the 3D image acquisition unit 110A-2, the angles acquired by the angle acquisition unit 120A-1, and the angles acquired by the angle acquisition unit 120A-2. The combination processing unit 130A acquires the angle θ between the optical axis 250-1 of the main lens unit 211-1 and the optical axis 250-2 of the main lens unit 211-2 from the angles acquired by the angle acquisition unit 120A-1 and the angle acquired by the angle acquisition unit 120A-2. The combination processing unit 130A uses the angle θ to generate a three-dimensional image acquired by the three-dimensional image acquisition unit 110A-1 or a diagonal image of the three-dimensional image acquired by the three-dimensional image acquisition unit 110A-2 (step ST1161 in FIG. 13). The combination processing unit 130A calculates pixel values ​​for each subdivided voxel using the three-dimensional image acquired by the three-dimensional image acquisition unit 110A-1 or the three-dimensional image acquired by the three-dimensional image acquisition unit 110A-2 and the diagonal image (step ST1162 in FIG. 13).

[0044] The image output device 10A then executes a composite 3D image output process (step ST1170). In the composite 3D image output process, the combination processing unit 130A of the image output device 10A reconstructs the composite 3D image using the pixel values ​​of each subdivided voxel. The combination processing unit 130A outputs the generated composite 3D image to an external device outside the image output device 10. The combination processing unit 130A outputs the composite 3D image to, for example, an external display device.

[0045] After executing the composite 3D image output process (step ST1170), the image output device 10 proceeds to an end determination process (step ST1180). In the end determination process, a control unit (not shown) of the image output device 10 determines whether to end the process of the image output device 10. The control unit (not shown) determines whether to end the process of the image output device 10, for example, in accordance with an external end command or an execution program. If the control unit (not shown) determines not to end the process of the image output device 10 ("NO" in step ST1180), the process proceeds to step ST1140, and the process is repeated from step ST1140. If the control unit (not shown) determines to end the process of the image output device 10 ("YES" in step ST1180), the image output device 10 terminates the process.

[0046] In this embodiment, the following configuration is disclosed: An image output device that outputs a three-dimensional image using images captured through a lens group having a main lens and a lens array, the image output device including: a three-dimensional image reconstruction unit that outputs a composite three-dimensional image using images captured through two sets of lens groups whose optical axes intersect with each other, and the angle formed by the optical axes of the two sets of lens groups. This has the effect of providing an image output device that can improve the depth resolution of a three-dimensional image in light field imaging.

[0047] In this embodiment, the following configuration is disclosed: An image output method by an image output device that outputs a three-dimensional image using images captured through a lens group having a main lens and a lens array, the image output method including a three-dimensional image reconstructing step in which a three-dimensional image reconstruction unit of the image output device outputs a composite three-dimensional image using images captured through two sets of the lens groups whose optical axes intersect with each other and the angle formed by the optical axes of the two sets of lens groups. This advantageously enables the present disclosure to provide an image output method that can improve the depth resolution of a three-dimensional image in light field imaging.

[0048] This embodiment further discloses the following configuration. The image output device is characterized in that the 3D image reconstruction unit includes: a 3D image acquisition unit that acquires, for each of the lens groups, a 3D image constructed using images captured through the lens group; an angle acquisition unit that acquires the angle between the optical axes of the two lens groups; and a combination processing unit that outputs the composite 3D image, which is an image combining the 3D images acquired by the 3D image acquisition unit and the angle acquired by the angle acquisition unit. This advantageous effect of the present disclosure is that it is possible to provide an image output device that can improve the depth resolution of a 3D image using an existing light field imaging system. Furthermore, the advantageous effect of the present disclosure is that it is possible to provide an image output device that performs processing to improve the depth resolution of a 3D image at a location separate from the location where the image is captured. Furthermore, the aforementioned configuration according to the present disclosure can achieve the same advantageous effect when applied to the aforementioned image output method.

[0049] This embodiment further discloses the following configuration: an image output device including: a plurality of lens groups, each having the main lens and the lens array; a plurality of imaging units, each of which captures an image through the lens group and outputs image data; and a plurality of 3D image construction units, each of which captures an image using the images captured by the imaging units, wherein the 3D image acquisition unit acquires the 3D image constructed by the 3D image construction unit. This provides an advantage that the present disclosure can provide an integrated image output device including an optical system. Furthermore, when the configuration according to the present disclosure is applied to the image output method, the same advantage as the above can be achieved.

[0050] Embodiment 2. Embodiment 2 describes a first example combination of the focal lengths of two sets of main lenses. In this embodiment, among the components according to embodiment 2, duplicated descriptions of the components similar to the components according to embodiment 1 already described will be omitted as appropriate.

[0051] An example configuration of an image output device according to a second embodiment of the present disclosure and an image output system including the device is similar to the example configuration already described. In the image output device according to the second embodiment, a case will be described in which the focal lengths of the main lens units 211 (211-1, ..., 211-n (2 ≧ n)) in the lens groups 210 (210-1, ..., 210-n (2 ≧ n)) of the two sets of LFIS 200 (200-1, ..., 200-n (2 ≧ n)) are far from the main lens units 211 (211-1, ..., 211-n (2 ≧ n)). FIG. 14 is a diagram illustrating an image of voxels after processing by the image output device according to the second embodiment of the present disclosure. FIG. 14 illustrates a three-dimensional space (an image of data representing the three-dimensional space) used by the image output device as viewed from above (as viewed toward the arrow β in FIG. 2).

[0052] 14, when the focal lengths of the main lens units 211 (211-1, ..., 211-n (2≧n)) are far from the main lens units 211 (211-1, ..., 211-n (2≧n)), the number of voxels (number of three-dimensional pixels) increases by 15 times in areas far from the main lens. Also, the number of voxels (number of three-dimensional pixels) increases by 6 times in areas close to the main lens units 211 (211-1, ..., 211-n (2≧n)).

[0053] In this embodiment, the three-dimensional image construction unit 230-N (N = 1, 2, ..., n (2 ≦ n)) performs a refocusing process from the focal position of the main lens unit 211 (211-1, ..., 211-n (2 ≧ n)) toward the main lens unit 211 (211-1, ..., 211-n (2 ≧ n)) to construct a three-dimensional image.

[0054] This embodiment also discloses the following configuration: an image output device, wherein the 3D image constructing unit constructs a 3D image by performing a refocusing process from the focal position of the main lens toward the main lens. This provides an advantage of providing an image output device that can improve resolution, particularly in areas far from the main lens. Furthermore, when the configuration according to the present disclosure is applied to the image output method, the same advantage as the above can be achieved.

[0055] Embodiment 3. Embodiment 3 describes a second example combination of the focal lengths of the two sets of main lenses. In this embodiment, among the components according to embodiment 3, duplicated descriptions of components similar to those according to embodiment 1 or embodiment 2 already described will be omitted as appropriate.

[0056] An example configuration of an image output device according to a third embodiment of the present disclosure and an image output system including the device is similar to the example configuration already described. In the image output device according to the third embodiment, the focal lengths of the main lens units 211 (211-1, ..., 211-n (2 ≧ n)) in the lens groups 210 (210-1, ..., 210-n (2 ≧ n)) of the two sets of LFIS 200 (200-1, ..., 200-n (2 ≧ n)) are close to the main lens units 211 (211-1, ..., 211-n (2 ≧ n)). FIG. 15 is a diagram illustrating an image of voxels after processing by the image output device according to the third embodiment of the present disclosure. FIG. 15 illustrates a three-dimensional space (an image of data representing the three-dimensional space) used by the image output device as viewed from above (as viewed toward the arrow β in FIG. 2).

[0057] 15, when the focal lengths of the main lens units 211 (211-1, ..., 211-n (2≧n)) are close to the main lens units 211 (211-1, ..., 211-n (2≧n)), the number of voxels (number of three-dimensional pixels) increases by three times in areas far from the main lens units 211 (211-1, ..., 211-n (2≧n)). Also, the number of voxels (number of three-dimensional pixels) increases by 17 times in areas close to the main lens units.

[0058] In this embodiment, for example, one of the three-dimensional image construction units 230 (230-1) of the three-dimensional image construction units 230-N (N = 1, n (2 ≦ n)) performs a refocusing process from the focal position of the main lens unit 211 (211-1) in the direction opposite to the direction of the main lens unit 211 (211-1) (direction away from the main lens unit 211 (211-1)) to construct a three-dimensional image. For example, one of the three-dimensional image construction units 230-N (N = 1, n (2 ≦ n)) performs a refocusing process from the focal position of the main lens unit 211 (211-2) in the direction opposite to the direction of the main lens unit 211 (211-2) (direction away from the main lens unit 211 (211-2)) to construct a three-dimensional image.

[0059] This embodiment also discloses the following configuration: an image output device, wherein the 3D image constructing unit constructs a 3D image by performing a refocusing process from the focal position of the main lens in a direction opposite to the direction of the main lens. This provides an advantage of providing an image output device that can improve resolution, particularly in areas close to the main lens. Furthermore, when the configuration according to the present disclosure is applied to the image output method, the same advantage as the above can be achieved.

[0060] Embodiment 4. Embodiment 4 describes a third example combination of the focal lengths of the two sets of main lenses. In this embodiment, among the components according to embodiment 4, duplicated descriptions of components similar to those according to embodiment 1, embodiment 2, or embodiment 3 already described will be omitted as appropriate.

[0061] An example configuration of an image output device according to a fourth embodiment of the present disclosure and an image output system including the device is similar to the example configuration already described. In the image output device according to the fourth embodiment, a case will be described in which the focal lengths of the main lens units 211 (211-1, ..., 211-n (2 ≧ n)) in the lens groups 210 (210-1, ..., 210-n (2 ≧ n)) of two sets of LFIS 200 (200-1, ..., 200-n (2 ≧ n)) are combined, with one being far from the main lens units 211 (211-1, ..., 211-n (2 ≧ n)) and the other being close to the main lens units 211 (211-1, ..., 211-n (2 ≧ n)). FIG. 16 is a diagram illustrating an image of voxels after processing by the image output device according to the fourth embodiment of the present disclosure. FIG. 16 illustrates a three-dimensional space (an image of data representing the three-dimensional space) used by the image output device of the present disclosure as viewed from above.

[0062] As shown in FIG. 16, when the focal lengths of the main lens units 211 (211-1, 211-n (2 ≧ n)) in the lens groups 210 (210-1, 210-n (2 ≧ n)) of the two LFIS 200 (200-1, 200-n (2 ≧ n)) are combined, one that is far from the main lens unit 211 (211-1, 211-n (2 ≧ n)) and one that is close to the main lens unit 211 (211-1, 211-n (2 ≧ n)), the number of voxels (number of three-dimensional pixels) is increased by 11 times in the region far from the main lens unit 211 (211-1, 211-n (2 ≧ n)). Also, the number of voxels (number of three-dimensional pixels) is increased by 12 times in the region close to the main lens unit 211 (211-1, ..., 211-n (2 ≧ n)). This embodiment has the feature that the resolution in the depth direction is improved over the entire observation range by deliberately calibrating the lenses to match two different focal planes.

[0063] In this embodiment, if the two lens groups 210 (210-1, 210-n (2≧n)) are a first lens group 210 (e.g., 210-1) and a second lens group 210 (e.g., 210-2), the length between the position of the main lens unit 211 (e.g., 211-1) in the first lens group 210 (e.g., 210-1) and the focal position of the main lens unit 211 (e.g., 211-2) is configured to be shorter than the length between the position of the main lens unit 211 (e.g., 211-2) in the second lens group 210 (e.g., 210-2) and the focal position of the main lens unit 211 (e.g., 211-2). Among the multiple three-dimensional image construction units 230-N (N = 1, ..., n (2 ≦ n)), at least one three-dimensional image construction unit 230 (e.g., 230-1), namely a first three-dimensional image construction unit 230 (e.g., 230-1), performs a refocusing process from the focal position of the main lens unit 211 (e.g., 211-1) in the first lens group 210 (e.g., 210-1) toward the main lens unit 211 (e.g., 211-1) to construct a three-dimensional image. Among the multiple three-dimensional image construction units 230-N (N = 1, ..., n (2 ≦ n)), the second three-dimensional image construction unit 230 (e.g., 230-2), which is at least one three-dimensional image construction unit 230 (e.g., 230-2) different from the first three-dimensional image construction unit 230 (e.g., 230-1), performs a refocusing process from the focal position of the main lens unit 211 (e.g., 211-2) in the second lens group 210 (e.g., 210-2) in a direction opposite to the direction of the main lens unit 211 (e.g., 211-2) (a direction away from the main lens unit 211 (e.g., 211-2)) to construct a three-dimensional image.

[0064] In the present embodiment, the following configuration is further disclosed: when the two lens groups are a first lens group and a second lens group, the distance between the position of a main lens in the first lens group and the focal position of the main lens is shorter than the distance between the position of a main lens in the second lens group and the focal position of the main lens, and the 3D image construction unit is configured such that, among the plurality of 3D image construction units, at least one 3D image construction unit, a first 3D image construction unit, constructs a 3D image by performing a refocusing process from the focal position of the main lens in the first lens group toward the main lens, and at least one 3D image construction unit, a second 3D image construction unit different from the first 3D image construction unit, constructs a 3D image by performing a refocusing process from the focal position of the main lens in the second lens group toward the main lens. This provides an advantageous effect of providing an image output device with improved depth resolution throughout the entire observation range. Furthermore, when the configuration according to the present disclosure is applied to the image output method, the same effect as the above effect is achieved.

[0065] Embodiment 5. Embodiment 5 describes an embodiment in which the direction of the optical axis of the main lens is variable. In this embodiment, among the components of embodiment 5, duplicated descriptions of components similar to those of embodiment 1, embodiment 2, embodiment 3, or embodiment 4 already described will be omitted as appropriate.

[0066] Next, a configuration example of a system including an image output device according to this embodiment will be described. Fig. 17 is a diagram showing a configuration example of an image output device according to embodiment 5 of the present disclosure and an image output system to which the image output device is applied. Fig. 18 is a diagram showing an image of an implementation example of the image output device and image output system according to embodiment 5 of the present disclosure. In addition to at least one of the features already described, the image output system 1E further has a function of changing the optical axis.

[0067] The image output system 1E shown in FIG. 17 includes an image output device 10E, LFIS (light field imaging systems) 200 (200-1, 200-2, ..., 200-n (2≦n)), a shooting synchronization unit 300, a light source 400, an angle control unit 500 (500-1, 500-2, ..., 500-n (2≦n)), and an angle designation mechanism 510 (510-1, 510-2, ..., 510-n (2≦n)).

[0068] The image output device 10E is configured in a manner similar to the image output devices 10 and 10A already described. The LFIS 200 (200-1, 200-2, ..., 200-n (2≦n)) is configured to include a combination of a lens group 210-N (N=1, 2, ..., n (2≦n)), an imaging unit 220-N (N=1, 2, ..., n (2≦n)), and a three-dimensional image construction unit 230-N (N=1, 2, ..., n (2≦n)), similar to the LFIS 200 already described (200-1, 200-2, ..., 200-n (2≦n)). The image synchronization unit 300 is configured in a manner similar to the image synchronization unit 300 already described. The light source 400 is configured in a manner similar to the light source 400 already described. The following description will focus on the content not described above.

[0069] The angle designation mechanism 510 (510-1, 510-2, ..., 510-n (2≦n)) changes the angle of the lens group 210 (210-1, 210-n (2≦n)) to a designated angle for each lens group 210 (210-1, 210-n (2≦n)), thereby changing the direction of the optical axis 250 (250-1, 250-n (2≦n)) of the main lens portion 211 (211-1, 211-n (2≦n)) in the lens group 210 (210-1, 210-n (2≦n)).

[0070] The angle control units 500 (500-1, 500-2, ..., 500-n (2≦n)) control the operation of the angle designation mechanisms 510 (510-1, 510-2, ..., 510-n (2≦n)). The angle control units 500 (500-1, 500-2, ..., 500-n (2≦n)) may be configured to be included inside the angle designation mechanisms 510 (510-1, 510-2, ..., 510-n (2≦n)).

[0071] The angle control units 500 (500-1, 500-2, ..., 500-n (2≦n)) and the angle designation mechanisms 510 (510-1, 510-2, ..., 510-n (2≦n)) may be capable of controlling a three-dimensional solid angle, or may be capable of controlling a planar angle. Alternatively, the angle control units 500 (500-1, 500-2, ..., 500-n (2≦n)) and the angle designation mechanisms 510 (510-1, 510-2, ..., 510-n (2≦n)) may be capable of determining the relative angular deviation with respect to the LFIS 200-1, or vice versa, and it does not matter if either one is tilted. This is because what is important in the technology of the present disclosure is the angle between the two sets of optical axes that intersect with each other.

[0072] The image output device 10E will be described below, focusing on the details not mentioned in the description of the image output devices 10 and 10A. The image output device 10E shown in Fig. 17 includes a three-dimensional image reconstruction unit 100E.

[0073] The three-dimensional image reconstruction unit 100E includes a three-dimensional image acquisition unit 110E (110E-1, 110E-2, ..., 110E-n (2≦n)), an angle acquisition unit 120E (120E-1, 120E-2, ..., 120E-n (2≦n)), and a combination processing unit 130E. The three-dimensional image acquisition units 110E (110E-1, ..., 110E-n (2≦n)) are similar to the three-dimensional image acquisition units 110 (110-1, ..., 110-n (2≦n)) and 110A (110A-1, ..., 110A-n (2≦n)) already described. The combination processing unit 130E is similar to the combination processing units 130 and 130A already described.

[0074] The angle acquisition unit 120E (120E-1, ..., 120E-n (2≦n)) acquires the angle θ formed by the optical axes of the two lens groups from the angle designation mechanism 510. Alternatively, the angle acquisition unit 120E (120E-1, ..., 120E-n (2≦n)) acquires the angle related to the optical axis designated by the angle designation mechanism 510 from the angle control unit 500. That is, the angle formed by the optical axes of the two lens groups 210 (210-1, 210-n (2≦n)) is an angle obtained from the angle designation mechanism 510, which changes the direction of the optical axis 250 (250-1, 250-n (2≦n)) of the main lens portion 211 (211-1, 211-n (2≦n)) in the lens group 210 (210-1, 210-n (2≦n)) by changing the angle of the lens group 210 (210-1, 210-n (2≦n)) to a designated angle for each lens group 210 (210-1, 210-n (2≦n)).

[0075] A processing example of an image output device or an image output system according to Embodiment 5 of the present disclosure will be described. Fig. 19 is a flowchart illustrating an example of processing of an image output device or an image output system according to Embodiment 5 of the present disclosure. The processing illustrated in Fig. 19 is an image output method by the image output system or the image output device.

[0076] The processing shown in Fig. 19 will be described in detail. In the following description, it is assumed that two sets of lens groups of LFIS 200-N (N = 1, 2, ... n (2 < n)), which is a combination of lens groups 210-N (N = 1, 2, ... n (2 < n)), imaging unit 220-N (N = 1, 2, ... n (2 < n)), and three-dimensional image construction unit 230-N (N = 1, 2, ... n (2 < n)), are lens groups 210-1 and 210-2, and that two sets of LFIS are LFIS 200-1 and 200-2. The image output system 1E or image output device 10E starts processing when, for example, a control unit (not shown) of image output device 10E receives an image output command.

[0077] The image output system 1E or the image output device 10E then executes angle control processing. (Step ST5110) In the angle control processing, the angle control unit 500 (500-1, 500-2) controls the operation of the angle designation mechanism 510 (510-1, 510-2). The angle control unit 500 (500-1, 500-2), for example, accepts an operation from a user and determines an angle based on the user operation. The angle control unit 500 (500-1, 500-2) controls the operation of the angle designation mechanism 510 (510-1, 510-2) based on the determined angle. The angle control unit 500 (500-1, 500-2) or the angle designation mechanism 510 (510-1, 510-2) outputs the angle of the optical axis based on the operation of the angle designation mechanism 510 (510-1, 510-2) to the angle acquisition unit 120E (120E-1, 120E-2).

[0078] The image output system 1E or the image output device 10E then executes a shooting synchronization process (step ST5120). In the shooting synchronization process, the shooting synchronization unit 300 outputs a TTL signal to the two sets of LFIS 200-1 and LFIS 200-2, which synchronizes shooting in the two sets of LFIS 200-1 and LFIS 200-2 among the combinations of the lens group 210-N (N=1, 2, ... n (2≦n)), the imaging unit 220-N (N=1, 2, ... n (2≦n)), and the three-dimensional image construction unit 230-N (N=1, 2, ... n (2≦n)).

[0079] Next, the image output system 1E or the image output device 10E executes an imaging data acquisition process. (Step ST5130) In the imaging data acquisition process, the multiple imaging units 220 (220-1, 220-2) capture images via the lens group 210 (210-1, 210-2) for each lens group 210 (210-1, 210-2) and output the imaging data. Here, the imaging unit 220-1 captures images via the lens group 210-1 and outputs the imaging data to the three-dimensional image construction unit 230-1. Also, the imaging unit 220-2 captures images via the lens group 210-2 and outputs the imaging data to the three-dimensional image construction unit 230-2.

[0080] Image output system 1E or image output device 10E then executes three-dimensional image construction processing. (Step ST5140) In the three-dimensional image construction processing, three-dimensional image construction unit 230 (230-1, 230-2) constructs a three-dimensional image for each imaging unit 220 (220-1, 220-2) using images captured by imaging unit 220 (220-1, 220-2). Here, three-dimensional image construction unit 230-1 constructs a three-dimensional image using images captured by imaging unit 220-1 and outputs the constructed three-dimensional image to three-dimensional image acquisition unit 110E-1. Furthermore, three-dimensional image construction unit 230-2 constructs a three-dimensional image using images captured by imaging unit 220-2 and outputs the constructed three-dimensional image to three-dimensional image acquisition unit 110E-2.

[0081] The image output device 10E then executes a three-dimensional image acquisition process (step ST5150). In the three-dimensional image acquisition process, the three-dimensional image acquisition units 110E (110E-1, 110E-2, ..., 110E-n (n≧2)) of the image output device 10E acquire, for example, three-dimensional images output for each of the plurality of LFISs 200 (200-1, 200-2). The three-dimensional image acquisition units 110E (110E-1, 110E-2) acquire, for each of the lens groups 210 (210-1, 210-2), three-dimensional images constructed using images captured via the lens groups 210 (210-1, 210-2). The three-dimensional image acquisition units 110E (110E-1, 110E-2, ..., 110E-n (n≧2)) each output the acquired three-dimensional images to the combination processing unit 130E. Three-dimensional image acquisition unit 110E-1 acquires a three-dimensional image constructed using images captured via lens group 210-1. Three-dimensional image acquisition unit 110E-2 acquires a three-dimensional image constructed using images captured via lens group 210-2.

[0082] The image output device 10E then executes angle acquisition processing (step ST5160). In the angle acquisition processing, the angle acquisition units 120E (120E-1, 120E-2) of the image output device 10E acquire the angle formed by the optical axes of two lens groups (e.g., two lens groups formed by the combination of lens group 200-1 and lens group 200-2). The angle acquisition units 120E (120E-1, 120E-2) each acquire the angle related to the optical axis or the angle formed by the intersecting optical axes from the angle control units 500 (500-1, 500-2) or the angle designation mechanisms 510 (510-1, 510-2). The angle acquisition units 120E (120E-1, 120E-2) each output the acquired angle to the combination processing unit 130. Alternatively, the angle acquisition unit 120E (120E-1, 120E-2) outputs to the combination processing unit 130 the angle θ formed between the optical axis 250-1 of the main lens unit 211-1 and the optical axis 250-2 of the main lens unit 211-2.

[0083] The image output device 10E then executes a combination process (step ST5170). In the combination process, the combination processing unit 130E of the image output device 10E uses the 3D images acquired by the 3D image acquisition units 110E (110E-1 and 110E-2) and the angles acquired by the angle acquisition units 120E (120E-1 and 120E-2) to output a composite 3D image, which is an image obtained by combining the 3D images. Specifically, the combination processing unit 130E acquires, for example, the 3D image acquired by the 3D image acquisition unit 110E-1, the 3D image acquired by the 3D image acquisition unit 110E-2, the angles acquired by the angle acquisition unit 120E-1, and the angles acquired by the angle acquisition unit 120E-2. The combination processing unit 130E acquires the angle θ between the optical axis 250-1 of the main lens unit 211-1 and the optical axis 250-2 of the main lens unit 211-2 from the angle acquired by the angle acquisition unit 120E-1 and the angle acquired by the angle acquisition unit 120E-2. The combination processing unit 130E uses the angle θ to generate a three-dimensional image acquired by the three-dimensional image acquisition unit 110E-1 or an oblique image of the three-dimensional image acquired by the three-dimensional image acquisition unit 110E-2 (similar to step ST1161 already described). The combination processing unit 130E calculates a pixel value for each subdivided voxel using the three-dimensional image acquired by the three-dimensional image acquisition unit 110E-1 or the three-dimensional image acquired by the three-dimensional image acquisition unit 110E-2 and the oblique image (similar to step ST1162 already described).

[0084] The image output device 10E then executes a composite 3D image output process. (Step ST5180) In the composite 3D image output process, the combination processing unit 130E of the image output device 10E generates a composite 3D image by reconstructing the image using pixel values ​​for each subdivided voxel. The combination processing unit 130E outputs the generated composite 3D image to an external device from the image output device 10E. The combination processing unit 130E outputs the composite 3D image to, for example, an external display device.

[0085] After executing the composite 3D image output process (step ST5180), the image output device 10E proceeds to an end determination process (step ST5190). In the end determination process, a control unit (not shown) of the image output device 10E determines whether to end the process of the image output device 10E. The control unit (not shown) determines whether to end the process of the image output device 10E, for example, in accordance with an external end command or an execution program. If the control unit (not shown) determines not to end the process of the image output device 10E ("NO" in step ST5190), the process proceeds to step ST5150 and repeats the process from step ST5150. If the control unit (not shown) determines to end the process of the image output device 10E ("YES" in step ST5190), the image output device 10E ends the process.

[0086] This embodiment further discloses the following configuration: An image output device, wherein the angle formed by the optical axes of the two lens groups is an angle acquired from an angle designation mechanism that changes the direction of the optical axis of the main lens in each lens group by changing the angle of the lens group to a designated angle. This provides an advantage that the present disclosure can provide an image output device capable of outputting a higher-resolution three-dimensional image by changing the angle of the optical axis. It also provides an advantage that the present disclosure can provide an image output device capable of fixing the center of the optical axis at a specific angle (e.g., angle θ or angle φ). Furthermore, when the configuration according to the present disclosure is applied to the image output method, it provides the same advantage as the above.

[0087] Sixth Embodiment In the sixth embodiment, a composite three-dimensional image is output using three or more three-dimensional images. In the sixth embodiment, among the components according to the sixth embodiment, the same components as those according to the first, second, third, fourth, or fifth embodiment already described will not be described again.

[0088]

[0073] An example configuration of an image output device according to a sixth embodiment of the present disclosure and an image output system including the device will be described. FIG. 20 is a diagram illustrating an example configuration of an image output device according to the sixth embodiment of the present disclosure and an image output system to which the image output device is applied. FIG. 21 is a diagram illustrating an example implementation of the image output device and image output system according to the sixth embodiment of the present disclosure. An image output system 1F reconstructs a composite 3D image using three or more 3D images. The image output system 1F illustrated in FIG. 20 includes an image output device 10F, light field imaging systems (LFIS) 200 (200-1, 200-2, ..., 200-n (2≦n)), an imaging synchronization unit 300, angle control units 500 (500-1, 500-2, ..., 500-n (2≦n)), an angle designation mechanism 510 (510-1, 510-2, ..., 510-n (2≦n)), and a processing order command unit 600.

[0089] The image output device 10F is configured in substantially the same manner as the image output devices 10, 10A, and 10E already described. Similarly to the LFIS 200 (200-1, 200-2, ..., 200-n (2≦n)), each of the LFISs 200 (200-1, 200-2, ..., 200-n (2≦n)) includes a combination of a lens group 210-N (N=1, 2, ..., n (2≦n)), an imaging unit 220-N (N=1, 2, ..., n (2≦n)), and a three-dimensional image construction unit 230-N (N=1, 2, ..., n (2≦n)). The shooting synchronization unit 300 is configured in the same manner as the shooting synchronization unit 300 already described. The angle control units 500 (500-1, 500-2, ..., 500-n (2≦n)) are configured in the same manner as the angle control units 500 (500-1, 500-2, ..., 500-n (2≦n)) already described. The angle designation mechanisms 510 (510-1, 510-2, ..., 510-n (2≦n)) are configured in the same manner as the angle designation mechanisms 510 (510-1, 510-2, ..., 510-n (2≦n)) already described. In FIG. 21, a light source 400 is further provided. The light source 400 is configured in the same manner as the light source 400 already described. The following description will focus on the content not explained up to this point.

[0090] When the image output system 1E or the image output device 10E includes three or more (3≦n) sets of LFIS 200 (200-1, 200-2, ..., 200-n), the processing order command unit 600 selects two sets of LFIS 200 (200-1, 200-n (2≦n)) based on a predetermined rule to perform imaging. At this time, the processing order command unit 600 commands the three-dimensional image reconstruction unit 100F (100F-1, 100F-n (2≦n)) to start processing corresponding to the two sets of LFIS 200 (200-1, 200-n (2≦n)) to perform imaging. At this time, the processing order command unit 600 may issue a command to the angle control units 500 (500-1, 500-n (2≦n)) corresponding to the two sets of LFIS 200 (200-1, 200-n (2≦n)) to be imaged.

[0091] In this case, the angle formed by the optical axes 250-N (N=1, n(2≦n)) of the two lens groups 210-N (N=1, n(2≦n)) in the two LFISs 200 (200-1, 200-n(2≦n)) is configured so that, as shown in FIG. 21, one combination (LFIS200-1) out of three or more combinations (LFIS) is used as a reference, and the angle (θ, Φ) is the angle formed by the optical axis (250-1) of the lens group (210-1) in the reference combination (LFIS200-1) and the optical axes (250-2, 250-3) of the lens groups (210-2, 210-3) in the other combinations (LFIS200-2, LFIS200-3).

[0092] A processing example of an image output device or an image output system according to Embodiment 6 of the present disclosure will be described. Fig. 22 is a flowchart showing an example of processing of the image output device or the image output system according to Embodiment 6 of the present disclosure. Fig. 23 is a flowchart showing an example of processing order command processing of the image output device or the image output system according to Embodiment 6 of the present disclosure. The processing shown in Fig. 22 or Fig. 23 is an image output method by the image output system 1F or the image output device 10F.

[0093] The processing shown in Figure 22 will be described in detail. Below, it is assumed that of the LFIS 200-N (N = 1, 2, ... n (3 < n)), which is a combination of lens groups 210-N (N = 1, 2, ... n (3 < n)), imaging unit 220-N (N = 1, 2, ... n (3 < n)), and 3D image construction unit 230-N (N = 1, 2, ... n (3 < n)), two sets of lens groups are lens groups 210-1, 210-2 or lens groups 210-1, 210-3, and that two sets of LFIS are LFIS 200-1, 200-2 or LFIS 200-1, 200-3. The image output system 1F or image output device 10F starts processing when, for example, a control unit (not shown) of image output device 10F receives an image output command.

[0094] The image output system 1F or the image output device 10F then executes a processing order command process (step ST6110). In the processing order command process, the processing order command unit 600 first executes a combination information acquisition process (step ST6111 shown in FIG. 23). In the combination information acquisition process, the processing order command unit 600 acquires combination information, for example, from a storage unit (not shown). The combination information is information that can identify the processing order of two sets of LFIS (200-1, 200-2, or 200-1, 200-3) among the LFIS 200-N (N=1, 2, ... n (3≦n)). For example, the combination information is a combination of numbers or alphanumeric characters that indicate the processing order. The processing order command unit 600 then executes a processing order determination process (step ST6112 shown in FIG. 23). In the processing order determination process, the processing order command unit 600 determines two sets of LFIS (200-1, 200-2 or 200-1, 200-3) in the processing order based on the combination information. The processing order command unit 600 then executes a processing command in the processing order (step ST6113 shown in FIG. 23). In the processing command, the processing order command unit 600 commands, for example, the two sets of LFIS (200-1, 200-2) to perform imaging. The processing order command unit 600 commands, for example, the two sets of LFIS (200-1, 200-2) via the imaging synchronization unit 300, and then commands the two sets of LFIS (200-1, 200-3).

[0095] The image output system 1F or the image output device 10F then executes angle control processing. (Step ST6120) In the angle control processing, the angle control unit 500 (500-1, 500-2) controls the operation of the angle designation mechanism 510 (510-1, 510-2). The angle control unit 500 (500-1, 500-2), for example, accepts an operation from a user and determines an angle based on the user operation. The angle control unit 500 (500-1, 500-2) controls the operation of the angle designation mechanism 510 (510-1, 510-2) based on the determined angle. The angle control unit 500 (500-1, 500-2) or the angle designation mechanism 510 (510-1, 510-2) outputs the angle of the optical axis based on the operation of the angle designation mechanism 510 (510-1, 510-2) to the angle acquisition unit 120F (120F-1, 120F-2). The angle control unit 500 (500-1, 500-2) or the angle designation mechanism 510 (510-1, 510-2) outputs the angle θ of the optical axis 250-2 of the main lens unit 211-2 of the LFIS 200-2 relative to the optical axis 250-1 of the main lens unit 211-1 of the LFIS 200-1 to the angle acquisition unit 120F (120F-1, 120F-2), for example, using the LFIS 200-1 as a reference. The angle control unit 500 (500-1, 500-3) or the angle designation mechanism 510 (510-1, 510-3) outputs the angle Φ of the optical axis 250-3 of the main lens unit 211-3 of the LFIS 200-3 relative to the optical axis 250-1 of the main lens unit 211-1 of the LFIS 200-1 to the angle acquisition unit 120F (120F-1, 120F-2), for example, based on the LFIS 200-1.

[0096] The image output system 1F or the image output device 10F then executes a shooting synchronization process (step ST6130). In the shooting synchronization process, the shooting synchronization unit 300 outputs a TTL signal to the two sets of LFIS 200-1 and LFIS 200-2, which synchronizes shooting in the two sets of LFIS 200-1 and LFIS 200-2 among the combinations of the lens group 210-N (N=1, 2, ... n (2≦n)), the imaging unit 220-N (N=1, 2, ... n (2≦n)), and the 3D image construction unit 230-N (N=1, 2, ... n (2≦n)).

[0097] The image output system 1F or the image output device 10F then executes imaging data acquisition processing (step ST6140). In the imaging data acquisition processing, the multiple imaging units 220 (220-1, 220-2, ..., 220-n (n≧3)) capture images via the lens group 210 (210-1, 210-2, ..., 210-n (n≧3)) for each lens group 210 (210-1, 210-2, ..., 210-n (n≧3)) and output imaging data. Here, the imaging unit 220-1 captures images via the lens group 210-1 and outputs the imaging data to the three-dimensional image construction unit 230-1. Similarly, the imaging unit 220-2 captures images via the lens group 210-2 and outputs the imaging data to the three-dimensional image construction unit 230-2. Furthermore, the imaging section 220-n (n≧3) captures images via the lens group 210-n (n≧3) and outputs the captured image data to the three-dimensional image construction section 230-n (n≧3).

[0098] The image output system 1F or the image output device 10F then executes a three-dimensional image construction process (step ST6150). In the three-dimensional image construction process, the three-dimensional image construction unit 230 (230-1, 230-2, 230-n (n≧3)) constructs a three-dimensional image for each of the imaging units 220 (220-1, 220-2) using the images captured by the imaging units 220 (220-1, 220-2). Here, the three-dimensional image construction unit 230-1 constructs a three-dimensional image using the images captured by the imaging unit 220-1 and outputs the constructed three-dimensional image to the three-dimensional image acquisition unit 110F-1. Furthermore, the three-dimensional image construction unit 230-2 constructs a three-dimensional image using the images captured by the imaging unit 220-2 and outputs the constructed three-dimensional image to the three-dimensional image acquisition unit 110F-2. Furthermore, the three-dimensional image constructing unit 230-n (n≧3) constructs a three-dimensional image using the images captured by the imaging unit 220-n (n≧3) and outputs the image to the three-dimensional image acquiring unit 110F-n (n≧3).

[0099] The image output device 10F then executes a three-dimensional image acquisition process (step ST6160). In the three-dimensional image acquisition process, the three-dimensional image acquisition unit 110F (110F-1, 110F-2, ..., 110F-n (n≧3)) of the image output device 10F acquires, for example, three-dimensional images output for each of the plurality of LFISs 200 (200-1, 200-2). The three-dimensional image acquisition unit 110F (110F-1, 110F-2, ..., 110F-n (n≧3)) acquires a three-dimensional image constructed using images captured via the lens groups 210 (210-1, 210-2, ..., 210-n (n≧2)) for each of the lens groups 210 (210-1, 210-2, ..., 210-n (n≧2)). The three-dimensional image acquisition units 110F (110F-1, 110F-2, ..., 110F-n (n≧3)) each output the acquired three-dimensional images to the combination processing unit 130F. The three-dimensional image acquisition unit 110F-1 acquires a three-dimensional image constructed using images captured via the lens group 210-1. The three-dimensional image acquisition unit 110F-2 acquires a three-dimensional image constructed using images captured via the lens group 210-2. The three-dimensional image acquisition unit 110F-n (n≧3) acquires a three-dimensional image constructed using images captured via the lens group 210-n (n≧3).

[0100] The image output device 10F then executes angle acquisition processing (step ST6170). In the angle acquisition processing, the angle acquisition units 120F (120F-1, 120F-2) of the image output device 10F acquire the angle formed by the optical axes of two lens groups (e.g., two lens groups formed by the combination of lens group 200-1 and lens group 200-2). The angle acquisition units 120F (120F-1, 120F-2) each acquire the angle related to the optical axis or the angle formed by the intersecting optical axes from the angle control units 500 (500-1, 500-2) or the angle designation mechanisms 510 (510-1, 510-2). The angle acquisition units 120F (120F-1, 120F-2) each output the acquired angle to the combination processing unit 130. Alternatively, the angle acquisition unit 120F (120F-1, 120F-2) outputs to the combination processing unit 130 the angle θ formed between the optical axis 250-1 of the main lens unit 211-1 and the optical axis 250-2 of the main lens unit 211-2.

[0101] The image output device 10F then executes a combination process (step ST6180). In the combination process, the combination processing unit 130F of the image output device 10F uses the 3D images acquired by the 3D image acquisition units 110F (110F-1 and 110F-2) and the angles acquired by the angle acquisition units 120F (120F-1 and 120F-2) to output a composite 3D image, which is an image obtained by combining the 3D images. Specifically, the combination processing unit 130F acquires, for example, the 3D image acquired by the 3D image acquisition unit 110F-1, the 3D image acquired by the 3D image acquisition unit 110F-2, the angle acquired by the angle acquisition unit 120F-1, and the angle acquired by the angle acquisition unit 120F-2. The combination processing unit 130F acquires the angle θ between the optical axis 250-1 of the main lens unit 211-1 and the optical axis 250-2 of the main lens unit 211-2 from the angle acquired by the angle acquisition unit 120F-1 and the angle acquired by the angle acquisition unit 120F-2. The combination processing unit 130F uses the angle θ to generate a three-dimensional image acquired by the three-dimensional image acquisition unit 110F-1 or an oblique image of the three-dimensional image acquired by the three-dimensional image acquisition unit 110F-2 (similar to step ST1161 already described). The combination processing unit 130F calculates a pixel value for each subdivided voxel using the three-dimensional image acquired by the three-dimensional image acquisition unit 110F-1 or the three-dimensional image acquired by the three-dimensional image acquisition unit 110F-2 and the oblique image (similar to step ST1162 already described).

[0102] The image output device 10F then executes a composite 3D image output process (step ST6190). In the composite 3D image output process, the combination processing unit 130F of the image output device 10F reconstructs the composite 3D image using the pixel values ​​of each subdivided voxel. The combination processing unit 130F outputs the generated composite 3D image to an external device, such as an external display device.

[0103] After executing the composite 3D image output process (step ST6190), the image output device 10F proceeds to an end determination process (step ST6200). In the end determination process, a control unit (not shown) of the image output device 10F determines whether to end the process of the image output device 10E. The control unit (not shown) determines whether to end the process of the image output device 10F, for example, in accordance with an external end command or an execution program. If the control unit (not shown) determines not to end the process of the image output device 10F ("NO" in step ST6200), the process proceeds to step ST6160 and repeats the process from step ST6160. If the control unit (not shown) determines to end the process of the image output device 10F ("YES" in step ST6200), the image output device 10F ends the process.

[0104] This embodiment further discloses the following configuration: When the image output device includes three or more combinations of the lens groups, the imaging unit, and the three-dimensional image constructing unit, the angle formed by the optical axes of the two lens groups is an angle formed between the optical axes of the lens groups in one of the three or more combinations, based on the reference combination, and the optical axes of the lens groups in the other combinations. This provides an advantage of providing an image output device that can further improve resolution in the depth direction with a minimal configuration. Furthermore, when the configuration according to the present disclosure is applied to the image output method, the same advantage as the above can be achieved.

[0105] Here, a hardware configuration for realizing the functions of the present disclosure will be described. Fig. 24 is a diagram showing a first example of a hardware configuration for realizing the functions of the configuration of the present disclosure. Fig. 25 is a diagram showing a second example of a hardware configuration for realizing the functions of the configuration of the present disclosure. Parts of the image output systems 1A, 1E, and 1F of the present disclosure, and all or part of the image output devices 10, 10A, 10E, and 10F, are respectively realized by hardware such as those shown in Fig. 24 or Fig. 25.

[0106] As shown in Fig. 24, each of a part of the image output systems 1A, 1E, and 1F, and the whole or part of the image output devices 10, 10A, 10E, and 10F, is configured, for example, with a processor 10001, a memory 10002, an input / output interface 10003, and a communication circuit 10004. The processor 10001 and the memory 10002 are, for example, installed in a computer. The memory 10002 stores the computer, which includes three-dimensional image reconstruction units 100, 100A, 100E, and 100F, three-dimensional image acquisition units 110 (110-1, 110-2, ..., 110-n (n≧2)), 110A (110A-1, 110A-2, ..., 110A-n (n≧2)), 110E, and 110F, angle acquisition units 120 (120-1, 120-2, ..., 120-n (n≧2)), 120A (120A-1, 120A-2, ..., 120A-n (n≧2)), 120E, and 120F, combination processing units 130, 130A, 130E, and 130F, and three-dimensional image construction units 230 (230-1, 230-2, ..., 230-n (n≧2)), shooting synchronization unit 300, angle control unit 500 (500-1, 500-2, ..., 500-n (n≧2)), processing order command unit 600, or a control unit not shown in the figure. The processor 10001 reads out and executes the program stored in the memory 10002, whereby the three-dimensional image reconstruction units 100, 100A, 100E, 100F, the three-dimensional image acquisition units 110 (110-1, 110-2, ..., 110-n (n≧2)), 110A (110A-1, 110A-2, ..., 110A-n (n≧2)), 110E, 110F, the angle acquisition units 120 (120-1, 120-2, ..., 120-n (n≧2)), 120A (120A-1, 120A-2, ..., 120A-n (n≧2)), 120E, 120F, the combination processing units 130, 130A, 130E, 130F, the three-dimensional image construction unit 230 (230-1, 230-2, ..., 230-n (n≧2)), an imaging synchronization unit 300, an angle control unit 500 (500-1, 500-2, ..., 500-n (n≧2)), a processing order command unit 600, or a control unit not shown in the figure is realized.Furthermore, a storage unit (not shown) is realized by the memory 10002 or another memory (not shown). Furthermore, a communication unit (not shown) is realized by the communication circuit 10004.

[0107] The processor 10001 is, for example, a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a microprocessor, a microcontroller, or a DSP (Digital Signal Processor). The memory 10002 may be a non-volatile or volatile semiconductor memory such as RAM (Random Access Memory), ROM (Read Only Memory), EPROM (Erasable Programmable ROM), EEPROM (Electrically Erasable Programmable Read Only Memory), or flash memory, or a magnetic disk such as a hard disk or flexible disk, or an optical disk such as a CD (Compact Disc) or DVD (Digital Versatile Disc), or a magneto-optical disk. The processor 10001 and the memory 10002 or the communication circuit 10004 are connected in a state capable of transmitting data to each other. The processor 10001, memory 10002, and communication circuit 10004 are connected via an input / output interface 10003 so as to be capable of transmitting data to and from other hardware.

[0108] Or, in a part of the image output system 1A, 1E, 1F and the whole or part of the image output device 10, 10A, 10E, 10F, the three-dimensional image reconstruction unit 100, 100A, 100E, 100F, the three-dimensional image acquisition unit 110 (110-1, 110-2, ..., 110-n (n≧2)), 110A (110A-1, 110A-2, ..., 110A-n (n≧2)), 110E, 110F, the angle acquisition unit 120 (120-1, 120-2, ..., 120-n (n≧2)), 120A (120A-1, 120A-2, ..., 120A-n (n≧2)), 120E, 120F, and the combination processing unit At least one function of the units 130, 130A, 130E, 130F, the three-dimensional image construction unit 230 (230-1, 230-2, ..., 230-n (n≧2)), the imaging synchronization unit 300, the angle control unit 500 (500-1, 500-2, ..., 500-n (n≧2)), the processing order command unit 600, or a control unit not shown may be realized by a dedicated processing circuit 20001 as shown in FIG. 25.

[0109] The processing circuit 20001 may be, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), an FPGA (Field-Programmable Gate Array), a SoC (System-on-a-Chip), or a system LSI (Large-Scale Integration), etc. In addition, a storage unit (not shown) is realized by the memory 20002 or another memory (not shown). The memory 20002 may be a non-volatile or volatile semiconductor memory such as RAM (Random Access Memory), ROM (Read Only Memory), EPROM (Erasable Programmable ROM), EEPROM (Electrically Erasable Programmable Read Only Memory), or flash memory, or a magnetic disk such as a hard disk or flexible disk, or an optical disk such as a CD (Compact Disc) or DVD (Digital Versatile Disc), or a magneto-optical disk. The communication circuit 20004 implements a communication unit (not shown). The processing circuit 20001 and the memory 20002 or the communication circuit 20004 are connected in a state where they can transmit data to each other. In addition, the processing circuit 20001, the memory 20002, and the communication circuit 20004 are connected in a state where they can transmit data to other hardware via the input / output interface 20003.In addition, in a part of the image output system 1A, 1E, 1F and the whole or part of the image output device 10, 10A, 10E, 10F, the three-dimensional image reconstruction unit 100, 100A, 100E, 100F, the three-dimensional image acquisition unit 110 (110-1, 110-2, ..., 110-n (n≧2)), 110A (110A-1, 110A-2, ..., 110A-n (n≧2)), 110E, 110F, the angle acquisition unit 120 (120-1, 120-2, ..., 120-n (n≧2)), 120A (120A-1, 120A-2, ..., 120A-n (n≧2)), 120E, 120F, and the combination processing unit At least one function of the units 130, 130A, 130E, 130F, the three-dimensional image construction unit 230 (230-1, 230-2, ..., 230-n (n≧2)), the imaging synchronization unit 300, the angle control unit 500 (500-1, 500-2, ..., 500-n (n≧2)), the processing order command unit 600, or the control unit not shown may be realized by a separate processing circuit, or may be realized collectively by a processing circuit.

[0110] Or, in a part of the image output system 1A, 1E, 1F and the whole or part of the image output device 10, 10A, 10E, 10F, the three-dimensional image reconstruction unit 100, 100A, 100E, 100F, the three-dimensional image acquisition unit 110 (110-1, 110-2, ..., 110-n (n≧2)), 110A (110A-1, 110A-2, ..., 110A-n (n≧2)), 110E, 110F, the angle acquisition unit 120 (120-1, 120-2, ..., 120-n (n≧2)), 120A (120A-1, 120A-2, ..., 120A-n (n≧2)), 120E, 120F, and the combination processing unit 130, 130A, 130E, 130F, three-dimensional image construction unit 230 (230-1, 230-2, ..., 230-n (n≧2)), shooting synchronization unit 300, angle control unit 500 (500-1, 500-2, ..., 500-n (n≧2)), processing order command unit 600, or a control unit not shown, some of the functions may be realized by processor 10001 and memory 10002, and the remaining functions may be realized by processing circuit 20001.

[0111] It should be noted that, within the scope of this disclosure, the embodiments may be freely combined, any component of each embodiment may be modified, or any component of each embodiment may be omitted.

[0112] The present disclosure can improve the depth resolution of three-dimensional images in light field imaging, and is therefore suitable for use in three-dimensional image output devices, three-dimensional image output systems, and three-dimensional image output programs that utilize light field imaging technology.

[0113] 1A, 1E, 1F Image output system, 10, 10A, 10E, 10F Image output device, 100, 100A, 100E, 100F Three-dimensional image reconstruction unit, 110 (110-1, 110-2, ..., 110-n (n≧2)), 110A (110A-1, 110A-2, ..., 110A-n (n≧2)), 110E, 110F Three-dimensional image acquisition unit, 120 (120-1, 120-2, ..., 120-n (n≧2)), 120A (120A-1, 120A-2, ..., 120A-n (n≧2)), 120E, 120F Angle acquisition unit, 130, 130A, 130E, 130F Combination processing unit, 200 (200-1, 200-2, ..., 200-n (n ≧ 2)) LFIS (Light Field Imaging System), 210 (210-1, 210-2, ..., 210-n (n ≧ 2)) Lens group, 211 (211-1, 211-2, ..., 211-n (n ≧ 2)) Main lens unit, 212 (212-1, 212-2, ..., 212-n (n ≧ 2)) Array lens unit, 220 (220-1, 220-2, ..., 220-n (n ≧ 2)) Imaging unit, 230 (230-1, 230-2, ..., 230-n (n ≧ 2)) Three-dimensional image construction unit, 250 (250-1, 250-2, ..., 250-n (n ≧ 2)) Optical axis, 300, imaging synchronization unit, 400, light source, 500 (500-1, 500-2, ..., 500-n (n ≥ 2)), angle control unit, 510 (510-1, 510-2, ..., 510-n (n ≥ 2)), angle designation mechanism, 600, processing order command unit, 1000, three-dimensional space, 1010 (1010-1, 1010-2, ..., 1010-n (n ≥ 2)), Focal plane of the main lens, 1020 (1021 (1021-1, 1021-2, ..., 1021-n (n≧2)), 1022 (1022-1, 1022-2, ..., 1022-n (n≧2)), 1023 (1023-1, 1023-2, ..., 1023-n (n≧2)), 1024 (1024-1, 1024-2, ...・, 1024-n (n≧2)), 1025 (1025-1, 1025-2, ..., 1025-n (n≧2)), 1026 (1026-1, 1026-2, ... , 1026-n (n≧2)), 1027 (1027-1, 1027-2, ..., 1027-n (n≧2)), 1028 (1028-1, 1028-2, ...,1028-n (n≧2)), ...) Refocus plane, 1030 (1034, 1035) Voxel (three-dimensional pixel), 1100 Image capture object, 1110 Conventional three-dimensional image, 10001 Processor, 10002 Memory, 10003 Input / output interface, 10004 Communication circuit, 20001 Processing circuit, 20002 Memory, 20003 Input / output interface, 20004 Communication circuit.

Claims

1. An image output device that outputs a three-dimensional image using an image captured through a lens group having a main lens and a lens array, the image output device comprising: a three-dimensional image reconstruction unit that outputs a composite three-dimensional image using each image captured through each of two sets of the lens groups in a state where their optical axes intersect, and the angle of the angle formed by the optical axes of the two sets of lens groups.

2. The three-dimensional image reconstruction unit of the image output device according to claim 1, comprising: a three-dimensional image acquisition unit that acquires a three-dimensional image configured using the image captured through the lens group for each lens group; an angle acquisition unit that acquires the angle of the angle formed by the optical axes of each of the two sets of lens groups; and a combination processing unit that outputs the composite three-dimensional image, which is an image obtained by combining the three-dimensional images, using the three-dimensional image acquired by the three-dimensional image acquisition unit and the angle acquired by the angle acquisition unit.

3. The image output device according to claim 2, further comprising: a plurality of lens groups each having the main lens and the lens array; a plurality of imaging units that output imaging data by imaging through the lens group for each lens group; and a plurality of three-dimensional image configuration units that configure a three-dimensional image using the image captured by the imaging unit for each imaging unit, wherein the three-dimensional image acquisition unit acquires the three-dimensional image configured by the three-dimensional image configuration unit.

4. The image output device according to any one of claims 1 to 3, wherein the three-dimensional image configuration unit performs a refocusing process from the focal position of the main lens toward the main lens to configure a three-dimensional image.

5. The image output device according to any one of claims 1 to 3, wherein the three-dimensional image configuration unit performs a refocusing process from the focal position of the main lens in a direction opposite to the direction of the main lens to configure a three-dimensional image.

6. When the two lens groups are the first lens group and the second lens group, the length between the position of the main lens and the focal position of the main lens in the first lens group is shorter than the length between the position of the main lens and the focal position of the main lens in the second lens group. Among the plurality of three-dimensional image forming units, at least one first three-dimensional image forming unit that is a three-dimensional image forming unit performs refocusing processing from the focal position of the main lens in the first lens group toward the main lens to form a three-dimensional image. At least one second three-dimensional image forming unit that is different from the first three-dimensional image forming unit performs refocusing processing in a direction opposite to the direction of the main lens from the focal position of the main lens in the second lens group to form a three-dimensional image. The image output device according to any one of claims 1 to 3.

7. The angle formed by the optical axes of the two lens groups is an angle obtained from an angle specifying mechanism that changes the direction of the optical axis of the main lens in the lens group by changing the angle of the lens group to a specified angle for each lens group. The image output device according to any one of claims 1 to 6, characterized in that.

8. When three or more combinations of the lens group, the imaging unit, and the three-dimensional image forming unit are provided, the angle formed by the optical axes of the two lens groups is the angle formed by the optical axis of the lens group in a combination used as a reference and the optical axis of the lens group in another combination among the three or more combinations. The image output device according to claim 3.

9. An image output method by an image output device that outputs a three-dimensional image using an image captured through a lens group having a main lens and a lens array, wherein a three-dimensional image reconstruction unit of the image output device outputs a synthesized three-dimensional image using each image captured through each of the two lens groups whose optical axes intersect each other and the angle formed by the optical axes of the two lens groups. An image output method comprising a three-dimensional image reconstruction step.

Images