Image output device and image output method

Abstract

Improve the depth resolution of three-dimensional images in light field imaging. [Solution] An image output device (10A) equipped with a three-dimensional image reconstruction unit (100A) that outputs a composite three-dimensional image using each three-dimensional image output by two light field imaging systems (200-1, 200-2) arranged with lens groups (210-1, 210-2) having a main lens and a lens array and their optical axes (250-1, 250-2) intersecting, and the angle (θ) at which the optical axes intersect.

Description

[Technical field]

[0001] The present disclosure relates to an image output technique for outputting a three-dimensional image. [Background technology]

[0002] The 3D imaging technology is based on the technology of restoring depth information based on parallax information from two or more images. Among the image output technologies that use this technology to output 3D images, there is a technology called light field imaging, which combines a main lens and a lens array to compose a 3D image using images captured by a single image sensor. For example, Patent Document 1 describes how a single image sensor captures multiple images via a main lens and a lenslet array (corresponding to the above-mentioned "lens array") for light field imaging. In light field imaging, the positions of multiple refocus planes are calculated using the distance from the main lens to the focal plane, and the spacing between the refocus planes increases exponentially as the distance from the focal plane of the main lens increases. These refocus planes define the depth length of a voxel, which is the smallest unit of three-dimensional space for constructing a three-dimensional image. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Special Publication No. 2016-511980 Summary of the Invention [Problem to be solved by the invention]

[0004] Conventional light field imaging had the problem that the depth resolution of 3D images was significantly degraded because the depth length of a voxel increases exponentially as it moves away from the focal plane of the main lens.

[0005] The present disclosure is intended to solve the above-mentioned problems, and aims to improve the depth resolution of three-dimensional images in light field imaging. [Means for solving the problem]

[0006] The image output device of the present disclosure is An image output device that outputs a three-dimensional image using an image captured through a lens group having a main lens and a lens array, a three-dimensional image reconstruction unit that outputs a composite three-dimensional image using each image captured through the two lens groups whose optical axes intersect with each other, and the angle between the optical axes of the two lens groups; Equipped with 、 The angle between the optical axes of the two lens groups is: the angle is obtained from an angle designation mechanism that changes the direction of the optical axis of the main lens in each of the lens groups by changing the angle of the lens group to a designated angle, It is characterized by This is what was done. Effect of the Invention

[0007] According to the present disclosure, it is possible to achieve an effect of improving the depth resolution of a three-dimensional image in light field imaging. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram illustrating an example of a basic configuration of an image output device according to a first embodiment of the present disclosure. [Diagram 2] FIG. 2 is a diagram showing an image of a three-dimensional space composed of voxels. [Diagram 3] FIG. 3 is a diagram showing an image of a refocus plane obtained by the refocus process. [Figure 4] FIG. 4 is a diagram showing a three-dimensional space formed by voxels as viewed from above. [Diagram 5]FIG. 5 is a diagram showing an image of a case where a three-dimensional image is obtained by a set of light field imaging system in which a lens group, an imaging section, and a three-dimensional image constructing section are combined. [Figure 6] FIG. 6 is a diagram showing an example of a three-dimensional image obtained by a conventional light field imaging system. [Figure 7] FIG. 7 is a diagram illustrating an image of the combination process according to the first embodiment of the present disclosure. [Figure 8] FIG. 8 is a flowchart illustrating an example of processing of the image output device according to the first embodiment of the present disclosure. [Figure 9] FIG. 9 is a diagram illustrating a configuration example in which the image output device according to the first embodiment of the present disclosure is applied to an image output system. [Figure 10] FIG. 10 is a diagram illustrating an image of a light field imaging system that outputs a three-dimensional image used in the image output device according to the first embodiment of the present disclosure. [Figure 11] FIG. 11 is a diagram illustrating an example of an implementation of the image output device and the image output system according to the first embodiment of the present disclosure. [Figure 12] FIG. 12 is a flowchart illustrating an example of processing of the image output device or the image output system according to the first embodiment of the present disclosure. [Figure 13] FIG. 13 is a flowchart illustrating an example of a combination process in the process of the image output device according to the first embodiment of the present disclosure. [Figure 14] FIG. 14 is a diagram showing an image of a voxel after processing by the image output device according to the second embodiment of the present disclosure. [Figure 15] FIG. 15 is a diagram showing an image of a voxel after processing by the image output device according to the third embodiment of the present disclosure. [Figure 16] FIG. 16 is a diagram showing an image of a voxel after processing by the image output device according to the fourth embodiment of the present disclosure. [Figure 17]FIG. 17 is a diagram illustrating a configuration example of an image output device according to the fifth embodiment of the present disclosure and an image output system to which the image output device is applied. [Figure 18] FIG. 18 is a diagram illustrating an example of an implementation of an image output device and an image output system according to the fifth embodiment of the present disclosure. [Figure 19] FIG. 19 is a flowchart illustrating an example of processing of the image output device or the image output system according to the fifth embodiment of the present disclosure. [Figure 20] FIG. 20 is a diagram illustrating a configuration example of an image output device according to the sixth embodiment of the present disclosure and an image output system to which the image output device is applied. [Figure 21] FIG. 21 is a diagram showing an image of an implementation example of an image output device and an image output system according to the sixth embodiment of the present disclosure. [Figure 22] FIG. 22 is a flowchart showing an example of processing of the image output device or image output system according to the sixth embodiment of the present disclosure. [Figure 23] FIG. 23 is a flowchart showing an example of a processing order command process of the image output device or image output system according to the sixth embodiment of the present disclosure. [Figure 24] FIG. 24 is a diagram illustrating a first example of a hardware configuration for realizing the functions according to the configuration of the present disclosure. [Diagram 25] FIG. 25 is a diagram illustrating a second example of a hardware configuration for realizing the functions according to the configuration of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] In order to explain the present disclosure in more detail, embodiments of the present disclosure will be described below with reference to the accompanying drawings. The embodiments of the present disclosure are intended to output three-dimensional images using light field imaging techniques. Furthermore, the embodiments of the present disclosure are based on the idea of ​​capturing images by overlapping two or more imaging areas by applying the configuration of an LFIS (Light Field Imaging System) equipped with two or more main lenses having different focal planes and lens arrays having different F-numbers. Furthermore, an embodiment of the present disclosure is based on the idea that by synthesizing a three-dimensional image from two light field images and using lateral pixels to compensate for the depth pixel size of the overlapping area, the lateral pixels correspond to the depth pixels, thereby improving the depth resolution.

[0010] Embodiment 1 In the first embodiment, a basic form of the present disclosure will be described.

[0011] An example of the configuration of an image output device according to the first embodiment of the present disclosure will be described. FIG. 1 is a diagram illustrating an example of a basic configuration of an image output device according to a first embodiment of the present disclosure. The image output device 10 outputs a three-dimensional image using images captured through a lens group having a main lens and a lens array. The image output device 10 shown in FIG.

[0012] The three-dimensional image reconstruction unit 100 outputs a composite three-dimensional image using each image captured through each of the two lens groups, whose optical axes intersect, and the angle formed by the optical axes of the two lens groups. The three-dimensional image reconstruction unit 100 shown in FIG. 1 is configured to include a three-dimensional image acquisition unit 110 (110-1, 110-2, ..., 110-n (2 <= n)), an angle acquisition unit 120 (120-1, 120-2, ..., 120-n (2 <= n)), and a combination processing unit 130.

[0013] Three-dimensional image acquisition unit 110 (110-1, 110-2, ..., 110-n (2 <= n)) acquires a three-dimensional image constructed using images captured through each lens group (for example, for each of two sets of lens groups formed by the combination of lens group 200-1 and lens group 200-2 described later). The three-dimensional image acquisition units 110 (110-1, 110-2, . . . , 110-n (2≦n)) acquire, for example, three-dimensional images output from each of a plurality of light field imaging systems (see FIGS. 9, 10, and 11, which will be described later). The multiple three-dimensional image acquisition units 110 (110-1, 110-2, . . . , 110-n (2≦n)) can also be collectively expressed as a "three-dimensional image acquisition unit group."

[0014] Angle acquisition units 120 (120-1, 120-2, . . . , 120-n (2 ≦ n)) acquire the angle between the optical axes of two lens groups (for example, two lens groups formed by a combination of lens group 200-1 and lens group 200-2 described below). The multiple angle acquisition units 120 (120-1, 120-2, . . . , 120-n (2 ≦ n)) can also be collectively referred to as an "angle acquisition unit group." For example, angle acquisition unit 120-1 or angle acquisition unit 120-2 acquires the angle formed by the optical axes of two lens groups (for example, two lens groups constituted by a combination of lens group 200-1 and lens group 200-2 described later). In addition, when the system is composed of only two lens groups (for example, when the two lens groups are composed of only a combination of lens group 200-1 and lens group 200-2 described below), it is sufficient to obtain the angle of the optical axis of one lens group based on the other lens group, so the system may have only one of angle acquisition unit 120-1 or angle acquisition unit 120-2.

[0015] The combination processing unit 130 combines a plurality of three-dimensional images to output a three-dimensional image (a composite three-dimensional image). The combination processing unit 130 outputs a composite three-dimensional image, which is an image combining the three-dimensional images, using the three-dimensional images acquired by the three-dimensional image acquisition unit 110 (110-1, 110-2, ..., 110-n (2 <= n)) and the angles acquired by the angle acquisition unit 120 (120-1, 120-2, ..., 120-n (2 <= n)). Specifically, when combining two three-dimensional images, the combination processing unit 130 uses the two three-dimensional images and the angle formed by the optical axes of each of the two lens groups (for example, two lens groups formed by the combination of lens group 200-1 and lens group 200-2 described below) to calculate pixel values ​​for each of the overlapping parts between the voxels of one three-dimensional image and the voxels of the other three-dimensional image, treating them as voxels. The combination processing unit 130 reconstructs a composite 3D image using the calculated pixel values ​​for each voxel.

[0016] In addition to the above components, the image output device 10 includes a control unit (not shown), a storage unit (not shown), and a communication unit (not shown). A control unit (not shown) controls the entire image output device 10 and each of its components. The control unit (not shown) starts up the image output device 10 in accordance with, for example, an external command. The control unit (not shown) also controls the state of the image output device 10 (operating state = start-up, shutdown, sleep, etc.). A storage unit (not shown) stores each data used by the image output device 10. The storage unit (not shown) stores, for example, the output (output data) by each component in the image output device 10, and outputs data requested by each component to the component that made the request. The communication unit (not shown) communicates with external devices or with internal components. For example, communication is performed between the image output device 10 (100A) and a peripheral device (e.g., a display device (not shown)). For example, when the image output device 10 and the peripheral device are not connected by wire, the communication unit (not shown) has a function of communicating between the image output device 10 and the peripheral device. The communication unit (not shown) also has a function of communicating between the image output device 10 and a server device (not shown). The server device may be configured to take on some of the functions of the image output device. A control unit (not shown), a storage unit (not shown), and a communication unit (not shown) each apply similarly to the embodiments described below.

[0017] Here, an overview of the processing performed by the image output device of the present disclosure will be described. FIG. 2 is a diagram showing an image of a three-dimensional space 1000 composed of voxels. FIG. 3 is a diagram showing an image of a refocus plane obtained by the refocus process. FIG. 4 is a diagram showing a three-dimensional space formed by voxels as viewed from above. FIG. 5 is a diagram showing an image of a case where a three-dimensional image is obtained by a set of light field imaging system in which a lens group, an imaging section, and a three-dimensional image constructing section are combined. FIG. 6 is a diagram showing an example of a three-dimensional image obtained by a conventional light field imaging system. FIG. 7 is a diagram illustrating an image of the combination process according to the first embodiment of the present disclosure. Three-dimensional space 1000 is refocused along the optical axis (axis extending in the direction of arrow α) of the main lens in the lens group, based on the position of focal plane 1010 of the main lens, to define refocus planes (1020, 1021, 1022, 1023, 1024, 1025, ...), and the depth direction of voxels in three-dimensional space 1000 is divided by these refocus planes (1020, 1021, 1022, 1023, 1024, 1025, ...). FIG. 4 shows a view in the direction indicated by the arrow β in FIGS. As shown in FIG. 4, a voxel 1030 (1034, 1035), which is a unit of a three-dimensional pixel, becomes longer in the depth direction (z direction) the further away it is from the focal plane 1010 of the main lens. Here, a case will be considered in which an image capturing target 1100 exists between a refocus plane 1026 and a refocus plane 1027 as shown in FIG. The image capture target 1100 is assumed to be an actual spherical object. 5. In the three central voxels shown in FIG. 5, pixel values ​​representing the imaged object 1100 are generated. However, since the voxel is long in the depth direction between the refocus plane 1026 and the refocus plane 1027, a three-dimensional image according to the length is constructed. In a conventional light field imaging system, a three-dimensional image 1110 as shown in FIG. 6 is output from pixel values ​​of voxels between the refocus plane 1026 and the refocus plane 1027 shown in FIG. In contrast, when a three-dimensional image is reconstructed according to the concept of the present disclosure, as shown in FIG. For example, the voxel unit can be expressed as 3-II and 2-II, and a three-dimensional pixel value can be expressed by a voxel that is finer than the conventional one voxel. By using three-dimensional pixel values ​​in units of voxels, it is possible to realize a representation closer to the real thing through higher resolution, and it is possible to increase the resolution of a three-dimensional image.

[0018] The process according to the first embodiment of the present disclosure will be described. FIG. 8 is a flowchart illustrating an example of processing performed by the image output device according to the first embodiment of the present disclosure.

[0019] The process shown in Fig. 8 is an image output method by an image output device that outputs a three-dimensional image using images captured through a lens group having a main lens and a lens array, and includes a three-dimensional image reconstruction step. In the three-dimensional image reconstruction step, a three-dimensional image reconstruction unit of the image output device outputs a composite three-dimensional image using each image captured through the two lens groups whose optical axes intersect with each other, and the angle formed by the optical axes of the two lens groups. The process shown in FIG. 8 will now be described in detail. The image output device 10 starts processing when, for example, a control unit (not shown) of the image output device 10 receives an image output command.

[0020] The image output device 10 then executes a three-dimensional image acquisition process (step ST110). In the three-dimensional image acquisition process, the three-dimensional image acquisition units 110 (110-1, 110-2, . . . , 110-n (2≦n)) of the image output device 10 acquire three-dimensional images output from, for example, each of a plurality of light field imaging systems. Three-dimensional image acquisition unit 110 (110-1, 110-2, ..., 110-n (2 <= n)) acquires a three-dimensional image constructed using images captured through each lens group (for example, for each of two sets of lens groups formed by the combination of lens group 200-1 and lens group 200-2 described later). Each of the three-dimensional image acquisition units 110 (110-1, 110-2, . . . , 110-n (2≦n)) outputs the acquired three-dimensional image to the combination processing unit .

[0021] The image output device 10 then executes an angle acquisition process (step ST120). In the angle acquisition process, the angle acquisition unit 120 (120-1, 120-2, ..., 120-n (2≦n)) of the image output device 10 acquires the angle between the optical axes of two lens groups (for example, two lens groups formed by the combination of lens group 200-1 and lens group 200-2 described later). Each of the angle acquisition units 120 (120-1, 120-2, . . . , 120-n (2≦n)) outputs the acquired angle to the combination processing unit 130.

[0022] The image output device 10 then executes a combination process (step ST130). In the combination process, the combination processing unit 130 of the image output device 10 outputs a composite three-dimensional image, which is an image combining the three-dimensional images, using the three-dimensional images acquired by the three-dimensional image acquisition unit 110 (110-1, 110-2, ..., 110-n (2 <= n)) and the angles acquired by the angle acquisition unit 120 (120-1, 120-2, ..., 120-n (2 <= n)). Specifically, the combination processing unit 130 acquires, for example, a three-dimensional image acquired by the three-dimensional image acquisition unit 110-1, a three-dimensional image acquired by the three-dimensional image acquisition unit 110-2, an angle acquired by the angle acquisition unit 120-1, and an angle acquired by the angle acquisition unit 120-2. The combination processing unit 130 combines the three-dimensional image acquired by the three-dimensional image acquisition unit 110-1 and the three-dimensional image acquired by the three-dimensional image acquisition unit 110-2 from the angle acquired by the angle acquisition unit 120-1 and the angle acquired by the angle acquisition unit 120-2, and generates and outputs a composite three-dimensional image reconstructed by the subdivided voxels.

[0023] The image output device 10 then executes a composite 3D image output process (step ST140). In the composite 3D image output process, the combination processing unit 130 of the image output device 10 outputs the generated composite 3D image to the outside of the image output device 10. The combination processing unit 130 outputs the composite 3D image to, for example, an external display device.

[0024] After executing the composite 3D image output process (step ST140), the image output device 10 proceeds to an end determination process (step ST150). In the end determination process, a control unit (not shown) of the image output device 10 determines whether to end the processing of the image output device 10. The control unit (not shown) determines whether to end the processing of the image output device 10 in accordance with, for example, an external end command or an execution program. When the control unit (not shown) determines not to end the processing of the image output device 10 ("NO" in step ST150), the process proceeds to step ST110, and the process is repeated from step ST110. When the control unit (not shown) determines that the processing of the image output device 10 is to be ended (step ST150 "YES"), the image output device 10 ends the processing.

[0025] Next, a configuration example of a system including an image output device according to the present embodiment will be described. FIG. 9 is a diagram illustrating a configuration example in which the image output device according to the first embodiment of the present disclosure is applied to an image output system. FIG. 10 is a diagram illustrating an image of a light field imaging system that outputs a three-dimensional image used in the image output device according to the first embodiment of the present disclosure. FIG. 11 is a diagram illustrating an example of an implementation of the image output device and the image output system according to the first embodiment of the present disclosure. The image output system 1A has at least two sets of main lenses having different focal planes and lens arrays having different F-numbers to match the main lenses. The image output system 1A also has a three-dimensional image reconstruction section 100A. The image output system 1A uses an LFIS (light field imaging system) equipped with two or more main lenses with different focal planes and lens arrays with different F-numbers to capture images by overlapping two or more imaging areas, thereby improving the resolution in the depth direction. An image output system 1A shown in FIG. 9 includes an image output device 10A, LFIS 200 (200-1, 200-2, . . . , 200-n (2≦n)), and a shooting synchronization section 300.

[0026] An image output system 1A shown in FIG. 9 includes a plurality of LFIS (light field imaging systems) 200 (200-1, 200-2, . . . , 200-n (2≦n)). Each of the LFIS 200 (200-1, 200-2,...,200-n (2≦n)) is configured to include a combination of a lens group 210-N (N=1, 2,...,n, 2≦n), an imaging unit 220-N (N=1, 2,...,n, 2≦n), and a three-dimensional image construction unit 230-N (N=1, 2,...,n, 2≦n). That is, the image output system 1A is configured to include a plurality of lens groups 210 (210-1, 210-2, ..., 210-n (2 <= n)), a plurality of imaging sections 220 (220-1, 220-2, ..., 220-n (2 <= n)), and a plurality of three-dimensional image construction sections 230 (230-1, 230-2, ..., 230-n (2 <= n)).

[0027] Lens group 210-N (N=1, 2,...,n, 2≦n) is configured to have a main lens portion 211-N (N=1, 2,...,n, 2≦n) and a lens array portion 212-N (N=1, 2,...,n, 2≦n). Lens group 210-1 shown in Fig. 9 has a main lens portion 211-1 and a lens array portion 212-1. Lens group 210-2 shown in Fig. 9 has a main lens portion 211-2 and a lens array portion 212-2. The main lens portion 211-N (N=1, 2, . . . , n, 2≦n) is composed of a lens (main lens). The lens array unit 212-N (N=1, 2, . . . , n, 2≦n) is composed of a lens array. That is, the multiple lens groups 210 (210-1, 210-2, ..., 210-n (2 <= n)) each have a main lens portion 211 (211-1, 211-2, ..., 211-n (2 <= n)) and a lens array portion 212 (212-1, 212-2, ..., 212-n (2 <= n)). The distance between the lens array of lens array section 212-N (N = 1, 2, ..., n, 2 <= n) and the imaging element of imaging section 220-N (N = 1, 2, ..., n, 2 <= n) is configured to be the same as the focal length of the lens array.

[0028] The imaging section 220-N (N=1, 2,...,n, 2≦n) is configured to include an imaging element, and outputs imaging data captured by the imaging element via the lens group 210-N (N=1, 2,...,n, 2≦n). That is, the multiple imaging units 220 (220-1, 220-2, ..., 220-n (2 <= n)) capture images via the lens group 210 (210-1, 210-2, ..., 210-n (2 <= n)) for each lens group 210 (210-1, 210-2, ..., 210-n (2 <= n)) and output imaging data. The multiple imaging units 220 (220-1, 220-2, . . . , 220-n (2≦n)) can also be collectively referred to as an "imaging unit group."

[0029] The main lens unit 211-N (N=1, 2,...., n, 2<=n), lens array unit 212-N (N=1, 2,...., n, 2<=n), and imaging unit 220-N (N=1, 2,...., n, 2<=n) are arranged in that order on the optical axis of the main lens in main lens unit 211-N (N=1, 2,...., n, 2<=n), and the image through the main lens is passed through the lens array to form the same number of images as the number of arrays, and is captured by the imaging element of imaging unit 220-N (N=1, 2,...., n, 2<=n).

[0030] Three-dimensional image constructing units 230-N (N=1, 2, . . . , n, 2≦n) construct a three-dimensional image using images captured by corresponding imaging units 220-N (N=1, 2, . . . , n, 2≦n). That is, multiple three-dimensional image construction units 230 (230-1, 230-2, ..., 230-n (2 <= n)) construct a three-dimensional image for each imaging unit 220 (220-1, 220-2, ..., 220-n (2 <= n)) using images captured by the imaging unit 220 (220-1, 220-2, ..., 220-n (2 <= n)). The multiple three-dimensional image constructing units 230 (230-1, 230-2, . . . , 230-n (2≦n)) can also be collectively expressed as a "three-dimensional image constructing unit group."

[0031] The imaging synchronization unit 300 synchronizes two pairs of LFIS 200 (200-1, 200-n (2≦n)) among the LFIS 200 (200-1, 200-2, . . . , 200-n (2≦n)). The shooting synchronization unit 300 is composed of, for example, a TTL signal generator, and outputs a TTL signal. The TTL signal generator may be any device that can continuously output a square wave with a specific timing. In the present disclosure, it is sufficient that the imaging by the two sets of LFIS 200 (200-1, 200-n (2≦n)) is synchronized, and the imaging synchronization unit 300 is not necessarily provided.

[0032] An image output device 10A shown in FIG. 9 has the same configuration as the image output device 10 already described. Regarding image output device 10A, the following mainly explains the points not mentioned in the explanation of image output device 10. An image output device 10A shown in FIG. 9 includes a three-dimensional image reconstruction section 100A. The three-dimensional image reconstruction unit 100A is configured to include a three-dimensional image acquisition unit 110A (110A-1, 110A-2, ..., 110A-n (2 <= n)), an angle acquisition unit 120A (120A-1, 120A-2, ..., 120A-n (2 <= n)), and a combination processing unit 130A. Three-dimensional image acquisition unit 110A (110A-1, 110A-2, ..., 110A-n (2 <= n)) acquires a three-dimensional image constructed by three-dimensional image construction unit 230 (230-1, 230-2, ..., 230-n (2 <= n)). Specifically, three-dimensional image acquisition unit 110A-1 acquires a three-dimensional image constructed by three-dimensional image construction unit 230-1. Three-dimensional image acquisition unit 110A-2 acquires a three-dimensional image constructed by three-dimensional image construction unit 230-2. Three-dimensional image acquisition unit 110A-n (2 <= n) acquires a three-dimensional image constructed by three-dimensional image construction unit 230-n (2 <= n). Note that the image output device 10A may be configured to include the range from that shown in Fig. 9 to that of the image output system 1A. For example, the image output device 10A may be configured to include all or a part of the multiple LFISs 200 (200-1, 200-2,..., 200-n (2 <= n)).

[0033] The two sets of LFIS 200-1, 200-2 are configured such that the focal lengths of the main lens units 211-1, 211-2 are different from each other, as shown in FIG. Here, the F-number of the lens array is expressed as in equation (1). F=f LA / P ···(1) "P" indicates the pitch of the lens array. "f LA " indicates the focal length of the lens array. When the two sets of LFIS 200 (200-1, 200-2) are configured so that the focal lengths of the main lens units 211 (211-1, 211-2) are different, the F-value F1 of the lens array unit 212-1 and the F-value F2 of the lens array unit 212-2 are different F-values, according to the following equation (1). However, the present disclosure is not limited to this, and as shown in an embodiment described later, the focal lengths of the main lens units 211 (211-1, 211-2) may be the same. The focal lengths may be far from the main lens units 211 (211-1, 211-2) or close to the main lens units 211 (211-1, 211-2).

[0034] FIG. 11 shows a configuration image of an image output system 1A or an image output device 10A in which the two sets of LFIS 200-1, 200-2 shown in FIG. 10 are arranged so that the optical axes 250-1, 250-2 of the main lens portions 211-1, 211-2 intersect with each other. If the angle θ is defined as the angle at which the optical axes intersect, the value D indicating the depth resolution is OL is expressed as the following formula (2), and it can be seen that the configuration of the present disclosure improves the resolution. D OL = 2d Lat (cos(θ / 2) / sinθ) (2) "d Lat " indicates the distance from the position of the main lens to the position of the focal plane. Or, the value D shown in the following formula (3) OL It can be seen that the resolution is increased. D OL = 2P(1-f LA / zM 2 ) ···(3) "z" indicates the distance from the position of the main lens to the position of the focal plane. "M" indicates the magnification. In this way, by using the data in the XY directions from the two sets of LFIS 200-1 and 200-2, it is possible to achieve high resolution.

[0035] The image output system 1A or the image output device 10A may be configured to include a light source 400 that irradiates light toward an imaging range or an imaging target as shown in Fig. 11. The light source 400 is, for example, a point light source. The light source 400 is, for example, a light source that irradiates diffuse light. The light source 400 is, for example, a light source that emits collimated light.

[0036] An example of processing in the image output system 1A or the image output device 10A will be described. FIG. 12 is a flowchart illustrating an example of processing of the image output device or the image output system according to the first embodiment of the present disclosure. FIG. 13 is a flowchart illustrating an example of a combination process in the process of the image output device according to the first embodiment of the present disclosure. The process shown in FIG. 12 and FIG. 13 is an image output method by an image output system or an image output device.

[0037] The process shown in Fig. 12 will be described in detail. In the following, it is assumed that two sets of lens groups are lens groups 210-1 and 210-2, and two sets of LFIS are LFIS 200-1 and 200-2, among LFIS 200-N (N = 1, 2,..., n (2 <= n)), which is a combination of lens group 210-N (N = 1, 2,..., n (2 <= n)), imaging unit 220-N (N = 1, 2,..., n (2 <= n)), and three-dimensional image construction unit 230-N (N = 1, 2,..., n (2 <= n)). The image output system 1A or the image output device 10A starts processing when, for example, a control unit (not shown) of the image output device 10A receives an image output command.

[0038] Next, the image output system 1A or the image output device 10A executes a shooting synchronization process (step ST1110). In the shooting synchronization processing, the shooting synchronization unit 300 outputs a TTL signal to the two LFISs 200-1 and 200-2, which synchronizes shooting in the two LFISs 200-1 and 200-2 among the combinations of the lens group 210-N (N=1, 2,...,n (2≦n)), the imaging unit 220-N (N=1, 2,...,n (2≦n)), and the three-dimensional image construction unit 230-N (N=1, 2,...,n (2≦n)).

[0039] Next, the image output system 1A or the image output device 10A executes an imaging data acquisition process (step ST1120). In the imaging data acquisition process, the imaging units 220 (220-1, 220-2) capture images via the lens groups 210 (210-1, 210-2) for each of the lens groups 210 (210-1, 210-2) and output imaging data. Here, imaging section 220-1 captures images via lens group 210-1 and outputs the captured image data to three-dimensional image construction section 230-1. Also, imaging section 220-2 captures images via lens group 210-2 and outputs the captured image data to three-dimensional image construction section 230-2.

[0040] Next, the image output system 1A or the image output device 10A executes a three-dimensional image construction process (step ST1130). In the three-dimensional image construction process, the three-dimensional image construction unit 230 (230-1, 230-2) constructs a three-dimensional image for each of the imaging units 220 (220-1, 220-2) using the images captured by the imaging units 220 (220-1, 220-2). Here, three-dimensional image construction section 230-1 constructs a three-dimensional image using images captured by imaging section 220-1 and outputs the image to three-dimensional image acquisition section 110A-1. Also, three-dimensional image construction section 230-2 constructs a three-dimensional image using images captured by imaging section 220-2 and outputs the image to three-dimensional image acquisition section 110A-2.

[0041] Next, the image output device 10A executes a three-dimensional image acquisition process (step ST1140). In the three-dimensional image acquisition process, three-dimensional image acquisition units 110A (110A-1, 110A-2, ..., 110A-n (n >= 2)) of image output device 10A acquire three-dimensional images output from, for example, each of a plurality of light field imaging systems. The three-dimensional image acquisition section 110 (110A-1, 110A-2) acquires, for each of the lens groups 210 (210-1, 210-2), a three-dimensional image constructed using images captured via the lens groups 210 (210-1, 210-2). Each of the three-dimensional image acquisition units 110A (110A-1, 110A-2, . . . , 110A-n (n≧2)) outputs the acquired three-dimensional image to the combination processing unit 130A. Three-dimensional image acquisition section 110A-1 acquires a three-dimensional image constructed using images captured via lens group 210-1. Three-dimensional image acquisition section 110A-2 acquires a three-dimensional image constructed using images captured via lens group 210-2.

[0042] Next, the image output device 10A executes an angle acquisition process (step ST1150). In the angle acquisition process, the angle acquisition unit 120A (120A-1, 120A-2) of the image output device 10A acquires the angle formed by the optical axes of two lens groups (for example, two lens groups formed by the combination of lens group 200-1 and lens group 200-2). Each of the angle acquisition units 120 (120-1, 120-2) outputs the acquired angle to the combination processing unit 130. Alternatively, the angle acquisition unit 120 (120-1, 120-2) combines and outputs to the processing unit 130 the angle θ formed between the optical axis 250-1 of the main lens unit 211-1 and the optical axis 250-2 of the main lens unit 211-2.

[0043] Next, the image output device 10A executes a combination process (step ST1160). In the combination process, the combination processing unit 130A of the image output device 10A outputs a composite three-dimensional image, which is an image obtained by combining the three-dimensional images, using the three-dimensional images acquired by the three-dimensional image acquisition unit 110A (110A-1, 110A-2) and the angles acquired by the angle acquisition unit 120A (120A-1, 120A-2). Specifically, the combination processing unit 130A acquires, for example, a three-dimensional image acquired by the three-dimensional image acquisition unit 110A-1, a three-dimensional image acquired by the three-dimensional image acquisition unit 110A-2, an angle acquired by the angle acquisition unit 120A-1, and an angle acquired by the angle acquisition unit 120A-2. The combination processing unit 130A acquires an angle θ between the optical axis 250-1 of the main lens unit 211-1 and the optical axis 250-2 of the main lens unit 211-2 from the angle acquired by the angle acquisition unit 120A-1 and the angle acquired by the angle acquisition unit 120A-2. The combination processing unit 130A uses the angle θ to generate an oblique image of the three-dimensional image acquired by the three-dimensional image acquiring unit 110A-1 or the three-dimensional image acquired by the three-dimensional image acquiring unit 110A-2 (step ST1161 in FIG. 13). The combination processing unit 130A calculates pixel values ​​for each subdivided voxel using the three-dimensional image acquired by the three-dimensional image acquisition unit 110A-1 or the three-dimensional image acquired by the three-dimensional image acquisition unit 110A-2 and the oblique image (step ST1162 in Figure 13).

[0044] Next, the image output device 10A executes a composite 3D image output process (step ST1170). In the composite 3D image output process, the combination processing unit 130A of the image output device 10A reconstructs the image using pixel values ​​of each subdivided voxel to generate a composite 3D image. The combination processing unit 130A outputs the generated composite 3D image to the outside of the image output device 10. The combination processing unit 130A outputs the composite 3D image to, for example, an external display device.

[0045] After executing the composite 3D image output process (step ST1170), the image output device 10 proceeds to an end determination process (step ST1180). In the end determination process, a control unit (not shown) of the image output device 10 determines whether to end the processing of the image output device 10. The control unit (not shown) determines whether to end the processing of the image output device 10 in accordance with, for example, an external end command or an execution program. When the control unit (not shown) judges not to end the processing of the image output apparatus 10 ("NO" in step ST1180), the process proceeds to step ST1140, and the process is repeated from step ST1140. When the control unit (not shown) determines that the processing of the image output device 10 is to be ended (step ST1180 "YES"), the image output device 10 ends the processing.

[0046] In this embodiment, the following configuration is disclosed. An image output device that outputs a three-dimensional image using an image captured through a lens group having a main lens and a lens array, a three-dimensional image reconstruction unit that outputs a composite three-dimensional image using each image captured through the two lens groups whose optical axes intersect with each other, and the angle between the optical axes of the two lens groups; Equipped with Image output device. As a result, the present disclosure has the effect of providing an image output device capable of improving the depth resolution of a three-dimensional image in light field imaging.

[0047] In this embodiment, the following configuration is disclosed. 1. An image output method by an image output device that outputs a three-dimensional image using an image captured through a lens group having a main lens and a lens array, comprising: a three-dimensional image reconstruction step in which a three-dimensional image reconstruction unit of the image output device outputs a composite three-dimensional image using each image captured through each of the two sets of lens groups whose optical axes intersect with each other, and the angle formed by the optical axes of the two sets of lens groups; Equipped with Image output method. As a result, the present disclosure has the effect of providing an image output method capable of improving the depth resolution of a three-dimensional image in light field imaging.

[0048] In this embodiment, the following configuration is further disclosed. The three-dimensional image reconstruction unit includes: a three-dimensional image acquisition unit that acquires, for each of the lens groups, a three-dimensional image constructed using images captured via the lens groups; an angle acquisition unit that acquires an angle between the optical axes of the two lens groups; a combination processing unit that uses the three-dimensional images acquired by the three-dimensional image acquisition unit and the angles acquired by the angle acquisition unit to output the composite three-dimensional image, which is an image obtained by combining the three-dimensional images; An image output device comprising: As a result, the present disclosure has an effect of providing an image output device capable of improving the depth resolution of a three-dimensional image by utilizing an existing light field imaging system. Also, the present disclosure has an effect of providing an image output device that executes a process for improving the depth resolution of a three-dimensional image at a location separate from the location where the image is captured. Furthermore, when the configuration according to the present disclosure is applied to the image output method, the same effect as the above effect is achieved.

[0049] In this embodiment, the following configuration is further disclosed. a plurality of the lens groups each including the main lens and the lens array; a plurality of imaging units configured to capture an image through each of the lens groups and output imaging data; A plurality of three-dimensional image constructing units each constructing a three-dimensional image using images captured by the imaging unit; Further equipped with The three-dimensional image acquisition unit acquires the three-dimensional image constructed by the three-dimensional image construction unit. 1. An image output device comprising: As a result, the present disclosure has an effect of providing an integrated image output device including an optical system. Furthermore, when the configuration according to the present disclosure is applied to the image output method, the same effect as the above effect is achieved.

[0050] Embodiment 2 In the second embodiment, a first example of a combination of focal lengths of two sets of main lenses will be described. In the second embodiment, among the components according to the second embodiment, the same components as those according to the first embodiment already described will not be described again.

[0051] The configuration example of the image output device according to the second embodiment of the present disclosure and the image output system including the device are similar to the configuration example already described. In the image output device according to the second embodiment, a case will be described in which the focal lengths of main lens units 211 (211-1,...,211-n (2>=n)) in lens groups 210 (210-1,...,210-n (2>=n)) of two sets of LFIS 200 (200-1,...,200-n (2>=n)) are far from the main lens units 211 (211-1,...,211-n (2>=n)). FIG. 14 is a diagram showing an image of a voxel after processing by the image output device according to the second embodiment of the present disclosure. FIG. 14 shows a three-dimensional space (an image of data showing a three-dimensional space) used by an image output device as viewed from above (as viewed in the direction of the arrow β in FIG. 2).

[0052] 14, when the focal lengths of the main lens units 211 (211-1, . . . , 211-n (2 ≧ n)) are far from the main lens units 211 (211-1, . . . , 211-n (2 ≧ n)), the number of voxels (number of three-dimensional pixels) in areas far from the main lens increases by 15 times. Also, the number of voxels (number of three-dimensional pixels) in areas close to the main lens units 211 (211-1, . . . , 211-n (2 ≧ n)) increases by 6 times.

[0053] In this embodiment, three-dimensional image construction unit 230-N (N = 1, 2, ... n (2 ≦ n)) performs a refocus process from the focal position of main lens unit 211 (211-1, ..., 211-n (2 ≧ n)) toward main lens unit 211 (211-1, ..., 211-n (2 ≧ n)) to construct a three-dimensional image.

[0054] In this embodiment, the following configuration is further disclosed. The three-dimensional image constructing unit constructs a three-dimensional image by performing a refocusing process from a focal position of the main lens toward the main lens. 1. An image output device comprising: As a result, the present disclosure has an effect of providing an image output device capable of improving resolution, particularly in areas far from the main lens. Furthermore, when the configuration according to the present disclosure is applied to the image output method, the same effect as the above effect is achieved.

[0055] Embodiment 3 In the third embodiment, a second combination example of the focal lengths of the two main lenses will be described. In the third embodiment, among the components of the third embodiment, those components which are similar to those of the first or second embodiment already described will not be described again.

[0056] The configuration example of the image output device according to the third embodiment of the present disclosure and the image output system including the device are similar to the configuration examples already described. In the image output device according to the third embodiment, a case will be described in which the focal lengths of main lens units 211 (211-1,...,211-n (2>=n)) in lens groups 210 (210-1,...,210-n (2>=n)) of two sets of LFIS 200 (200-1,...,200-n (2>=n)) are close to the main lens unit 211 (211-1,...,211-n (2>=n)). FIG. 15 is a diagram showing an image of a voxel after processing by the image output device according to the third embodiment of the present disclosure. FIG. 15 shows a three-dimensional space (an image of data showing a three-dimensional space) used by an image output device as viewed from above (as viewed in the direction of the arrow β in FIG. 2).

[0057] 15, when the focal lengths of the main lens units 211 (211-1,...,211-n (2>=n)) are close to the main lens units 211 (211-1,...,211-n (2>=n)), the number of voxels (number of three-dimensional pixels) increases threefold in areas far from the main lens units 211 (211-1,...,211-n (2>=n)). Also, the number of voxels (number of three-dimensional pixels) increases 17fold in areas close to the main lens.

[0058] In this embodiment, for example, one of the three-dimensional image construction units 230 (230-1) among the three-dimensional image construction units 230-N (N = 1, n (2 ≦ n)) performs a refocus process from the focal position of the main lens unit 211 (211-1) in the direction opposite to the direction of the main lens unit 211 (211-1) (direction away from the main lens unit 211 (211-1)) to construct a three-dimensional image. For example, three-dimensional image construction unit 230-2 among three-dimensional image construction units 230-N (N = 1, n (2 ≦ n)) performs a refocus process from the focal position of main lens unit 211 (211-2) in the opposite direction to the direction of main lens unit 211 (211-2) (direction away from main lens unit 211 (211-2)) to construct a three-dimensional image.

[0059] In this embodiment, the following configuration is further disclosed. The three-dimensional image constructing unit constructs a three-dimensional image by performing a refocusing process from a focal position of the main lens in a direction opposite to the direction of the main lens. 1. An image output device comprising: As a result, the present disclosure has an advantageous effect of providing an image output device capable of improving resolution, particularly in an area close to the main lens. Furthermore, when the configuration according to the present disclosure is applied to the image output method, the same effect as the above effect is achieved.

[0060] Embodiment 4 The fourth embodiment describes a third example of the combination of focal lengths of the two main lenses. In embodiment 4, among the components of embodiment 4, duplicated explanations will be omitted as appropriate for components similar to those of embodiment 1, embodiment 2, or embodiment 3 already described.

[0061] The configuration example of the image output device according to the fourth embodiment of the present disclosure and the image output system including the device are similar to the configuration examples already described. In the image output device according to the fourth embodiment, a case will be described in which the focal lengths of the main lens units 211 (211-1,...,211-n (2>=n)) in the lens groups 210 (210-1,...,210-n (2>=n)) of the two LFISs 200 (200-1,...,200-n (2>=n)) are combined, with one being far from the main lens unit 211 (211-1,...,211-n (2>=n)) and the other being close to the main lens unit 211 (211-1,...,211-n (2>=n)). FIG. 16 is a diagram showing an image of a voxel after processing by the image output device according to the fourth embodiment of the present disclosure. FIG. 16 shows a three-dimensional space (an image of data showing a three-dimensional space) used by the image output device of the present disclosure, viewed from above.

[0062] As shown in FIG. 16, when the focal lengths of the main lens units 211 (211-1, 211-n (2≧n)) in the lens groups 210 (210-1, 210-n (2≧n)) of two sets of LFIS 200 (200-1, 200-n (2≧n)) are combined, one that is far from the main lens unit 211 (211-1, 211-n (2≧n)) and one that is close to the main lens unit 211 (211-1, 211-n (2≧n)), the number of voxels (number of three-dimensional pixels) increases by 11 times in the area far from the main lens unit 211 (211-1, 211-n (2≧n)). Furthermore, in the region close to the main lens portion 211 (211-1, . . . , 211-n (2≧n)), the number of voxels (the number of three-dimensional pixels) increases by 12 times. This embodiment has the feature that the resolution in the depth direction is improved over the entire observation range by deliberately calibrating the lenses to match two different focal planes.

[0063] In this embodiment, if the two lens groups 210 (210-1, 210-n (2≧n)) are a first lens group 210 (e.g., 210-1) and a second lens group 210 (e.g., 210-2), the length between the position of the main lens portion 211 (e.g., 211-1) in the first lens group 210 (e.g., 210-1) and the focal position of the main lens portion 211 (e.g., 211-2) is configured to be shorter than the length between the position of the main lens portion 211 (e.g., 211-2) in the second lens group 210 (e.g., 210-2) and the focal position of the main lens portion 211 (e.g., 211-2). Among the multiple three-dimensional image construction units 230-N (N = 1, ..., n (2 <= n)), at least one three-dimensional image construction unit 230 (e.g., 230-1), i.e., a first three-dimensional image construction unit 230 (e.g., 230-1), performs a refocus process from the focal position of the main lens unit 211 (e.g., 211-1) in the first lens group 210 (e.g., 210-1) toward the main lens unit 211 (e.g., 211-1) to construct a three-dimensional image. Among the multiple three-dimensional image construction units 230-N (N = 1, ..., n (2 <= n)), a second three-dimensional image construction unit 230 (e.g., 230-2), which is at least one three-dimensional image construction unit 230 (e.g., 230-2) different from the first three-dimensional image construction unit 230 (e.g., 230-1), performs a refocus process from the focal position of the main lens unit 211 (e.g., 211-2) in the second lens group 210 (e.g., 210-2) in the opposite direction to the direction of the main lens unit 211 (e.g., 211-2) (direction away from the main lens unit 211 (e.g., 211-2)) to construct a three-dimensional image.

[0064] In this embodiment, the following configuration is further disclosed. wherein the two sets of lens groups are a first lens group and a second lens group; a distance between a position of the main lens in the first lens group and a focal position of the main lens is shorter than a distance between a position of the main lens in the second lens group and a focal position of the main lens, The three-dimensional image constructing unit includes: Among the plurality of three-dimensional image constructing units, The at least one three-dimensional image constructing unit, a first three-dimensional image constructing unit, A refocusing process is performed from a focal position of a main lens in the first lens group toward the main lens to form a three-dimensional image; A second three-dimensional image constructing unit which is at least one three-dimensional image constructing unit different from the first three-dimensional image constructing unit, performing a refocus process from a focal position of a main lens in the second lens group in a direction opposite to the direction of the main lens to form a three-dimensional image; 1. An image output device comprising: As a result, the present disclosure has an effect of providing an image output device that improves the resolution in the depth direction over the entire observation range. Furthermore, when the configuration according to the present disclosure is applied to the image output method, the same effect as the above effect is achieved.

[0065] Embodiment 5. In the fifth embodiment, a mode in which the direction of the optical axis of the main lens is variable will be described. In embodiment 5, among the components of embodiment 5, duplicated explanations will be omitted as appropriate for components similar to those of embodiment 1, embodiment 2, embodiment 3, or embodiment 4 already described.

[0066] Next, a configuration example of a system including an image output device according to the present embodiment will be described. FIG. 17 is a diagram illustrating a configuration example of an image output device according to the fifth embodiment of the present disclosure and an image output system to which the image output device is applied. FIG. 18 is a diagram illustrating an example of an implementation of an image output device and an image output system according to the fifth embodiment of the present disclosure. The image output system 1E has at least one of the features already described, and further has a function of changing the optical axis.

[0067] The image output system 1E shown in FIG. 17 is configured to include an image output device 10E, an LFIS (light field imaging system) 200 (200-1, 200-2, ..., 200-n (2 <= n)), a shooting synchronization unit 300, a light source 400, an angle control unit 500 (500-1, 500-2, ..., 500-n (2 <= n)), and an angle designation mechanism 510 (510-1, 510-2, ..., 510-n (2 <= n)).

[0068] The image output device 10E has a configuration similar to that of the image output devices 10 and 10A already described. The LFIS 200 (200-1, 200-2,...,200-n (2≦n)), like the LFIS 200 (200-1, 200-2,...,200-n (2≦n)) already described, is configured to include a combination of a lens group 210-N (N=1,2,...n (2≦n)), an imaging unit 220-N (N=1,2,...n (2≦n)), and a three-dimensional image construction unit 230-N (N=1,2,...n (2≦n)). The image synchronization unit 300 has the same configuration as the image synchronization unit 300 already described. The light source 400 is configured similarly to the light source 400 previously described. The following will mainly explain the points that have not been explained so far.

[0069] The angle designation mechanism 510 (510-1, 510-2, ..., 510-n (2 <= n)) changes the angle of the lens group 210 (210-1, 210-n (2 <= n)) to a designated angle for each lens group 210 (210-1, 210-n (2 <= n)), thereby changing the direction of the optical axis 250 (250-1, 250-n (2 <= n)) of the main lens portion 211 (211-1, 211-n (2 <= n)) in the lens group 210 (210-1, 210-n (2 <= n)).

[0070] Angle control units 500 (500-1, 500-2, . . . , 500-n (2 ≦ n)) control the operation of angle designation mechanisms 510 (510-1, 510-2, . . . , 510-n (2 ≦ n)). Angle control units 500 (500-1, 500-2, . . . , 500-n (2 ≦ n)) may be configured to be included inside angle designation mechanism 510 (510-1, 510-2, . . . , 510-n (2 ≦ n)).

[0071] Angle control unit 500 (500-1, 500-2, ..., 500-n (2 <= n)) and angle designation mechanism 510 (510-1, 510-2, ..., 510-n (2 <= n)) may be capable of controlling a three-dimensional solid angle, or may be capable of controlling a planar angle. Alternatively, it is sufficient for angle control units 500 (500-1, 500-2, ..., 500-n (2 <= n)) and angle designation mechanisms 510 (510-1, 510-2, ..., 510-n (2 <= n)) to be able to determine the relative angle deviation with respect to LFIS 200-1, or vice versa, and it is also acceptable for one of them to be tilted. This is because what is important in the technology of the present disclosure is the angle at which the two sets of optical axes intersect with each other.

[0072] Regarding image output device 10E, the following mainly describes the contents that have not been described in the description of image output devices 10 and 10A. An image output device 10E shown in FIG. 17 includes a three-dimensional image reconstruction section 100E.

[0073] The three-dimensional image reconstruction unit 100E is configured to include a three-dimensional image acquisition unit 110E (110E-1, 110E-2, ..., 110E-n (2 <= n)), an angle acquisition unit 120E (120E-1, 120E-2, ..., 120E-n (2 <= n)), and a combination processing unit 130E. Three-dimensional image acquisition unit 110E (110E-1,...,110E-n (2<=n)) is similar to the three-dimensional image acquisition units 110 (110-1,...,110-n (2<=n)) and 110A (110A-1,...,110A-n (2<=n)) already described. The combination processing unit 130E is similar to the combination processing units 130 and 130A already described.

[0074] Angle acquisition unit 120E (120E-1, . . . , 120E-n (2≦n)) acquires from angle designation mechanism 510 an angle θ formed by the optical axes of the two lens groups. Alternatively, the angle acquisition unit 120E (120E-1, . . . , 120E-n (2≦n)) acquires the angle related to the optical axis designated by the angle designation mechanism 510 from the angle control unit 500. In other words, the angle between the optical axes of the two lens groups 210 (210-1, 210-n (2≦n)) is an angle obtained from an angle designation mechanism 510 that changes the direction of the optical axis 250 (250-1, 250-n (2≦n)) of the main lens portion 211 (211-1, 211-n (2≦n)) in the lens group 210 (210-1, 210-n (2≦n)) by changing the angle of the lens group 210 (210-1, 210-n (2≦n)) to a specified angle for each lens group 210 (210-1, 210-n (2≦n)).

[0075] A processing example of an image output device or an image output system according to the fifth embodiment of the present disclosure will be described. FIG. 19 is a flowchart illustrating an example of processing of the image output device or the image output system according to the fifth embodiment of the present disclosure. The process shown in FIG. 19 is an image output method by an image output system or an image output device.

[0076] The process shown in Fig. 19 will be described in detail. In the following, it is assumed that two sets of lens groups are lens groups 210-1 and 210-2, and two sets of LFIS are LFIS 200-1 and 200-2, among LFIS 200-N (N=1,2,...n (2≦n)), which is a combination of lens group 210-N (N=1,2,...n (2≦n)), imaging unit 220-N (N=1,2,...n (2≦n)), and three-dimensional image construction unit 230-N (N=1,2,...n (2≦n)). The image output system 1E or the image output device 10E starts processing when, for example, a control unit (not shown) of the image output device 10E receives an image output command.

[0077] Next, the image output system 1E or the image output device 10E executes an angle control process (step ST5110). In the angle control process, the angle control unit 500 (500-1, 500-2) controls the operation of the angle designation mechanism 510 (510-1, 510-2). The angle control unit 500 (500-1, 500-2) receives, for example, an operation from a user and determines an angle based on the user's operation. The angle control unit 500 (500-1, 500-2) controls the operation of the angle designation mechanism 510 (510-1, 510-2) based on the determined angle. The angle control unit 500 (500-1, 500-2) or the angle designation mechanism 510 (510-1, 510-2) outputs the angle of the optical axis based on the operation of the angle designation mechanism 510 (510-1, 510-2) to the angle acquisition unit 120E (120E-1, 120E-2).

[0078] Next, the image output system 1E or the image output device 10E executes a shooting synchronization process (step ST5120). In the shooting synchronization process, the shooting synchronization unit 300 outputs a TTL signal to the two LFISs 200-1 and 200-2, which synchronizes shooting in the two LFISs 200-1 and 200-2 among the combinations of the lens group 210-N (N=1, 2,...n (2≦n)), the imaging unit 220-N (N=1, 2,...n (2≦n)), and the three-dimensional image construction unit 230-N (N=1, 2,...n (2≦n)).

[0079] Next, the image output system 1E or the image output device 10E executes an imaging data acquisition process (step ST5130). In the imaging data acquisition process, the imaging units 220 (220-1, 220-2) capture images via the lens groups 210 (210-1, 210-2) for each of the lens groups 210 (210-1, 210-2) and output imaging data. Here, imaging section 220-1 captures images via lens group 210-1 and outputs the captured image data to three-dimensional image construction section 230-1. Also, imaging section 220-2 captures images via lens group 210-2 and outputs the captured image data to three-dimensional image construction section 230-2.

[0080] The image output system 1E or the image output device 10E then executes a three-dimensional image construction process (step ST5140). In the three-dimensional image construction process, the three-dimensional image construction section 230 (230-1, 230-2) constructs a three-dimensional image for each of the imaging sections 220 (220-1, 220-2) using the images captured by the imaging sections 220 (220-1, 220-2). Here, the three-dimensional image construction unit 230-1 constructs a three-dimensional image using the images captured by the imaging unit 220-1 and outputs the image to the three-dimensional image acquisition unit 110E-1. The three-dimensional image construction unit 230-2 constructs a three-dimensional image using the images captured by the imaging unit 220-2 and outputs the image to the three-dimensional image acquisition unit 110E-2.

[0081] Image output device 10E then executes a three-dimensional image acquisition process (step ST5150). In the three-dimensional image acquisition process, three-dimensional image acquisition units 110E (110E-1, 110E-2, . . . , 110E-n (n≧2)) of image output device 10E acquire three-dimensional images output for each of the multiple LFISs 200 (200-1, 200-2), for example. The three-dimensional image acquisition section 110E (110E-1, 110E-2) acquires, for each of the lens groups 210 (210-1, 210-2), a three-dimensional image constructed using images captured via the lens groups 210 (210-1, 210-2). Each of the three-dimensional image acquisition units 110E (110E-1, 110E-2, . . . , 110E-n (n≧2)) outputs the acquired three-dimensional image to the combination processing unit 130E. Three-dimensional image acquisition section 110E-1 acquires a three-dimensional image constructed using images captured via lens group 210-1. Three-dimensional image acquisition section 110E-2 acquires a three-dimensional image constructed using images captured via lens group 210-2.

[0082] Image output device 10E then executes an angle acquisition process (step ST5160). In the angle acquisition process, the angle acquisition unit 120E (120E-1, 120E-2) of the image output device 10E acquires the angle formed by the optical axes of two lens groups (for example, two lens groups formed by the combination of lens group 200-1 and lens group 200-2). The angle acquisition units 120E (120E-1, 120E-2) each acquire an angle associated with an optical axis or an angle formed by the optical axes intersecting with each other from an angle control unit 500 (500-1, 500-2) or an angle designation mechanism 510 (510-1, 510-2). Each of the angle acquisition units 120E (120E-1, 120E-2) outputs the acquired angle to the combination processing unit . Alternatively, angle acquisition unit 120E (120E-1, 120E-2) combines and outputs to processing unit 130 angle θ formed between optical axis 250-1 of main lens unit 211-1 and optical axis 250-2 of main lens unit 211-2.

[0083] Image output device 10E then executes a combination process (step ST5170). In the combination process, the combination processing unit 130E of the image output device 10E outputs a composite three-dimensional image, which is an image obtained by combining the three-dimensional images, using the three-dimensional images acquired by the three-dimensional image acquisition unit 110E (110E-1, 110E-2) and the angles acquired by the angle acquisition unit 120E (120E-1, 120E-2). Specifically, the combination processing unit 130E acquires, for example, a three-dimensional image acquired by the three-dimensional image acquisition unit 110E-1, a three-dimensional image acquired by the three-dimensional image acquisition unit 110E-2, an angle acquired by the angle acquisition unit 120E-1, and an angle acquired by the angle acquisition unit 120E-2. The combination processing unit 130E acquires an angle θ between the optical axis 250-1 of the main lens unit 211-1 and the optical axis 250-2 of the main lens unit 211-2 from the angle acquired by the angle acquisition unit 120E-1 and the angle acquired by the angle acquisition unit 120E-2. The combination processing unit 130E uses the angle θ to generate an oblique image of the three-dimensional image acquired by the three-dimensional image acquisition unit 110E-1 or the three-dimensional image acquired by the three-dimensional image acquisition unit 110E-2 (similar to step ST1161 already described). The combination processing unit 130E calculates a pixel value for each divided voxel by using the three-dimensional image acquired by the three-dimensional image acquisition unit 110E-1 or the three-dimensional image acquired by the three-dimensional image acquisition unit 110E-2 and the oblique image (similar to step ST1162 described above).

[0084] Next, the image output device 10E executes a composite 3D image output process (step ST5180). In the composite 3D image output process, a combination processing unit 130E of the image output device 10E generates a composite 3D image by performing reconstruction using pixel values ​​for each subdivided voxel. The combination processing unit 130E outputs the generated composite 3D image to the outside of the image output device 10E. The combination processing unit 130E outputs the composite 3D image to, for example, an external display device.

[0085] After executing the composite 3D image output process (step ST5180), the image output device 10E proceeds to an end determination process (step ST5190). In the termination determination process, a control unit (not shown) of the image output device 10E determines whether to terminate the processing of the image output device 10E. The control unit (not shown) determines whether to terminate the processing of the image output device 10E in accordance with, for example, an external termination command or an execution program. When the control unit (not shown) determines not to end the process of the image output device 10E ("NO" in step ST5190), the process proceeds to step ST5150, and the process is repeated from step ST5150. If the control unit (not shown) determines to end the process of the image output device 10E ("YES" in step ST5190), the image output device 10E ends the process.

[0086] In this embodiment, the following configuration is further disclosed. The angle between the optical axes of the two lens groups is: the angle is obtained from an angle designation mechanism that changes the direction of the optical axis of the main lens in each of the lens groups by changing the angle of the lens group to a designated angle, 1. An image output device comprising: As a result, the present disclosure has the advantage of being able to provide an image output device capable of outputting a three-dimensional image with higher resolution by changing the angle of the optical axis, and also has the advantage of being able to provide an image output device capable of fixing the center of the optical axis at a specific angle (for example, angle θ or angle φ). Furthermore, when the configuration according to the present disclosure is applied to the image output method, the same effect as the above effect is achieved.

[0087] Embodiment 6 In the sixth embodiment, a configuration will be described in which a composite 3D image is output using three or more 3D images. In embodiment 6, among the components of embodiment 6, duplicated explanations will be omitted as appropriate for components similar to those of embodiment 1, embodiment 2, embodiment 3, embodiment 4, or embodiment 5 already described.

[0088] A configuration example of an image output device according to a sixth embodiment of the present disclosure and an image output system including the device will be described. FIG. 20 is a diagram illustrating a configuration example of an image output device according to the sixth embodiment of the present disclosure and an image output system to which the image output device is applied. FIG. 21 is a diagram showing an image of an implementation example of an image output device and an image output system according to the sixth embodiment of the present disclosure. The image output system 1F reconstructs a composite 3D image using three or more 3D images. The image output system 1F shown in FIG. 20 is configured to include an image output device 10F, an LFIS (light field imaging system) 200 (200-1, 200-2, ..., 200-n (2 <= n)), an imaging synchronization unit 300, an angle control unit 500 (500-1, 500-2, ..., 500-n (2 <= n)), an angle designation mechanism 510 (510-1, 510-2, ..., 510-n (2 <= n)), and a processing order command unit 600.

[0089] The image output device 10F is configured in a manner similar to the image output devices 10, 10A, and 10E already described. The LFIS 200 (200-1, 200-2,...,200-n (2≦n)), like the LFIS 200 (200-1, 200-2,...,200-n (2≦n)) already described, is configured to include a combination of a lens group 210-N (N=1,2,...n (2≦n)), an imaging unit 220-N (N=1,2,...n (2≦n)), and a three-dimensional image construction unit 230-N (N=1,2,...n (2≦n)). The image synchronization unit 300 has the same configuration as the image synchronization unit 300 already described. The angle control units 500 (500-1, 500-2, . . . , 500-n (2 ≦ n)) are configured similarly to the angle control units 500 (500-1, 500-2, . . . , 500-n (2 ≦ n)) already described. The angle designation mechanisms 510 (510-1, 510-2, . . . , 510-n (2 ≦ n)) are configured similarly to the angle designation mechanisms 510 (510-1, 510-2, . . . , 510-n (2 ≦ n)) already described. 21, a light source 400 is further provided. The light source 400 has the same configuration as the light source 400 already described. The following will mainly explain the points that have not been explained so far.

[0090] When the image output system 1E or the image output device 10E has three or more sets (3≦n) of LFIS 200 (200-1, 200-2, . . . , 200-n), the processing order command unit 600 selects two sets of LFIS 200 (200-1, 200-n (2≦n)) for imaging based on a predetermined rule. At that time, the processing order command unit 600 commands the three-dimensional image reconstruction unit 100F (100F-1, 100Fn (2≦n)) to start processing corresponding to the two sets of LFIS 200 (200-1, 200-n (2≦n)) to be imaged. At that time, the processing order command unit 600 may issue a command to the angle control units 500 (500-1, 500-n (2≦n)) corresponding to the two sets of LFIS 200 (200-1, 200-n (2≦n)) to be imaged.

[0091] In this case, the angle formed by the optical axes 250-N (N=1, n (2≦n)) of the two lens groups 210-N (N=1, n (2≦n)) in the two LFIS 200 (200-1, 200-n (2≦n)) is configured to be an angle (θ, Φ) formed by the optical axis (250-1) of the lens group (210-1) in one combination (LFIS200-1) of three or more combinations (LFIS) as a reference, and the optical axes (250-2, 250-3) of the lens groups (210-2, 210-3) in the other combinations (LFIS200-2, LFIS200-3) as a reference, as shown in FIG. 21 .

[0092] A processing example of an image output device or an image output system according to the sixth embodiment of the present disclosure will be described. FIG. 22 is a flowchart showing an example of processing of the image output device or image output system according to the sixth embodiment of the present disclosure. FIG. 23 is a flowchart showing an example of a processing order command process of the image output device or image output system according to the sixth embodiment of the present disclosure. The process shown in FIG. 22 or FIG. 23 is an image output method by the image output system 1F or the image output device 10F.

[0093] The process shown in Fig. 22 will be described in detail. In the following, it is assumed that of the LFIS 200-N (N = 1, 2,... n (3 < n)) which is a combination of the lens group 210-N (N = 1, 2,... n (3 < n)), the imaging unit 220-N (N = 1, 2,... n (3 < n)), and the three-dimensional image construction unit 230-N (N = 1, 2,... n (3 < n)), two sets of lens groups are the lens groups 210-1, 210-2 or the lens groups 210-1, 210-3, and that the two sets of LFIS are the LFIS 200-1, 200-2 or the LFIS 200-1, 200-3. The image output system 1F or the image output device 10F starts processing when, for example, a control unit (not shown) of the image output device 10F receives an image output command.

[0094] The image output system 1F or the image output apparatus 10F then executes a processing order command process (step ST6110). In the processing order command process, the processing order command unit 600 first executes a combination information acquisition process (step ST6111 shown in FIG. 23). In the combination information acquisition process, the processing order command unit 600 acquires combination information, for example, from a storage unit (not shown). The combination information is information that can identify the processing order of two sets of LFIS (200-1, 200-2, or 200-1, 200-3) among the LFIS 200-N (N=1, 2,...n (3≦n)). For example, the combination information is a combination of numbers or alphanumeric characters that indicate the processing order. The processing order command unit 600 then executes a processing order determination process (step ST6112 shown in FIG. 23). In the processing order determination process, the processing order command unit 600 determines two pairs of LFIS (200-1, 200-2 or 200-1, 200-3) in the processing order based on the combination information. The processing order command unit 600 then executes the processing command in the processing order (step ST6113 shown in FIG. 23). In the processing command, the processing order command unit 600 commands, for example, two sets of LFIS (200-1, 200-2) to perform imaging. The processing order command unit 600 commands, for example, the two sets of LFIS (200-1, 200-2) via the imaging synchronization unit 300, and then commands two sets of LFIS (200-1, 200-3).

[0095] Next, the image output system 1F or the image output device 10F executes an angle control process (step ST6120). In the angle control process, the angle control unit 500 (500-1, 500-2) controls the operation of the angle designation mechanism 510 (510-1, 510-2). The angle control unit 500 (500-1, 500-2) receives, for example, an operation from a user and determines an angle based on the user's operation. The angle control unit 500 (500-1, 500-2) controls the operation of the angle designation mechanism 510 (510-1, 510-2) based on the determined angle. The angle control unit 500 (500-1, 500-2) or the angle designation mechanism 510 (510-1, 510-2) outputs the angle of the optical axis based on the operation of the angle designation mechanism 510 (510-1, 510-2) to the angle acquisition unit 120F (120F-1, 120F-2). Angle control unit 500 (500-1, 500-2) or angle designation mechanism 510 (510-1, 510-2) outputs the angle θ of the optical axis 250-2 of the main lens unit 211-2 of the LFIS 200-2 relative to the optical axis 250-1 of the main lens unit 211-1 of the LFIS 200-1 to angle acquisition unit 120F (120F-1, 120F-2), for example, based on the LFIS 200-1. Angle control unit 500 (500-1, 500-3) or angle designation mechanism 510 (510-1, 510-3) outputs the angle Φ of the optical axis 250-3 of the main lens unit 211-3 of the LFIS 200-3 relative to the optical axis 250-1 of the main lens unit 211-1 of the LFIS 200-1 to angle acquisition unit 120F (120F-1, 120F-2), for example, based on the LFIS 200-1.

[0096] The image output system 1F or the image output device 10F then executes shooting synchronization processing (step ST6130). In the shooting synchronization process, the shooting synchronization unit 300 outputs a TTL signal to the two LFISs 200-1 and 200-2, which synchronizes shooting in the two LFISs 200-1 and 200-2 among the combinations of the lens group 210-N (N=1, 2,...n (2≦n)), the imaging unit 220-N (N=1, 2,...n (2≦n)), and the three-dimensional image construction unit 230-N (N=1, 2,...n (2≦n)).

[0097] The image output system 1F or the image output device 10F then executes an imaging data acquisition process (step ST6140). In the imaging data acquisition process, the multiple imaging units 220 (220-1, 220-2, ..., 220-n (n >= 3)) capture images via the lens group 210 (210-1, 210-2, ..., 210-n (n >= 3)) for each lens group 210 (210-1, 210-2, ..., 210-n (n >= 3)) and output imaging data. Here, the imaging section 220-1 captures images via the lens group 210-1 and outputs the captured image data to the three-dimensional image construction section 230-1. The imaging section 220-2 captures images via the lens group 210-2 and outputs the captured image data to the three-dimensional image construction section 230-2. The imaging section 220-n (n≧3) captures images via the lens group 210-n (n≧3) and outputs the captured image data to the three-dimensional image construction section 230-n (n≧3).

[0098] The image output system 1F or the image output device 10F then executes a three-dimensional image construction process (step ST6150). In the three-dimensional image construction process, three-dimensional image construction units 230 (230-1, 230-2, 230-n (n≧3)) construct a three-dimensional image for each of the imaging units 220 (220-1, 220-2) using images captured by the imaging units 220 (220-1, 220-2). Here, the three-dimensional image construction unit 230-1 constructs a three-dimensional image using images captured by the imaging unit 220-1 and outputs the image to the three-dimensional image acquisition unit 110F-1. The three-dimensional image construction unit 230-2 constructs a three-dimensional image using images captured by the imaging unit 220-2 and outputs the image to the three-dimensional image acquisition unit 110F-2. The three-dimensional image construction unit 230-n (n≧3) constructs a three-dimensional image using images captured by the imaging unit 220-n (n≧3) and outputs the image to the three-dimensional image acquisition unit 110F-n (n≧3).

[0099] Image output device 10F then executes a three-dimensional image acquisition process (step ST6160). In the three-dimensional image acquisition process, a three-dimensional image acquisition unit 110F (110F-1, 110F-2, . . . , 110F-n (n≧3)) of the image output device 10F acquires, for example, three-dimensional images output for each of the multiple LFISs 200 (200-1, 200-2). The three-dimensional image acquisition unit 110F (110F-1, 110F-2, ..., 110F-n (n≧3)) acquires a three-dimensional image constructed using images captured via the lens group 210 (210-1, 210-2, ..., 210-n (n≧2)) for each lens group 210 (210-1, 210-2, ..., 210-n (n≧2)). Each of the three-dimensional image acquisition units 110F (110F-1, 110F-2, . . . , 110F-n (n≧3)) outputs the acquired three-dimensional image to the combination processing unit 130F. The three-dimensional image acquisition section 110F-1 acquires a three-dimensional image constructed using images captured via the lens group 210-1. The three-dimensional image acquisition section 110F-2 acquires a three-dimensional image constructed using images captured via the lens group 210-2. Three-dimensional image acquisition section 110F-n (n≧3) acquires a three-dimensional image constructed using images captured via lens group 210-n (n≧3).

[0100] Next, the image output device 10F executes an angle obtaining process (step ST6170). In the angle acquisition process, the angle acquisition unit 120F (120F-1, 120F-2) of the image output device 10F acquires the angle formed by the optical axes of two lens groups (for example, two lens groups formed by the combination of lens group 200-1 and lens group 200-2). The angle acquisition units 120F (120E-1, 120F-2) each acquire an angle related to an optical axis or an angle formed by optical axes intersecting with each other from an angle control unit 500 (500-1, 500-2) or an angle designation mechanism 510 (510-1, 510-2). Each of the angle acquisition units 120F (120F-1, 120F-2) outputs the acquired angle to the combination processing unit . Alternatively, angle acquisition section 120F (120F-1, 120F-2) combines and outputs to processing section 130 angle θ formed between optical axis 250-1 of main lens section 211-1 and optical axis 250-2 of main lens section 211-2.

[0101] Next, the image output device 10F executes a combination process (step ST6180). In the combination process, the combination processing unit 130F of the image output device 10F outputs a composite three-dimensional image, which is an image combining the three-dimensional images, using the three-dimensional images acquired by the three-dimensional image acquisition unit 110F (110F-1, 110F-2) and the angles acquired by the angle acquisition unit 120F (120F-1, 120F-2). Specifically, the combination processing unit 130F acquires, for example, a three-dimensional image acquired by the three-dimensional image acquisition unit 110F-1, a three-dimensional image acquired by the three-dimensional image acquisition unit 110F-2, an angle acquired by the angle acquisition unit 120F-1, and an angle acquired by the angle acquisition unit 120F-2. The combination processing unit 130F acquires an angle θ between the optical axis 250-1 of the main lens unit 211-1 and the optical axis 250-2 of the main lens unit 211-2 from the angle acquired by the angle acquisition unit 120F-1 and the angle acquired by the angle acquisition unit 120F-2. The combination processing unit 130F uses the angle θ to generate an oblique image of the three-dimensional image acquired by the three-dimensional image acquisition unit 110F-1 or the three-dimensional image acquired by the three-dimensional image acquisition unit 110F-2 (similar to step ST1161 already described). The combination processing unit 130F calculates a pixel value for each divided voxel by using the three-dimensional image acquired by the three-dimensional image acquisition unit 110F-1 or the three-dimensional image acquired by the three-dimensional image acquisition unit 110F-2 and the oblique image (similar to step ST1162 described above).

[0102] The image output device 10F then executes a composite 3D image output process (step ST6190). In the composite 3D image output process, the combination processing unit 130F of the image output device 10F reconstructs the composite 3D image using the pixel values ​​of each subdivided voxel to generate the composite 3D image. The combination processing unit 130F outputs the generated composite 3D image to the outside of the image output device 10F. The combination processing unit 130F outputs the composite 3D image to, for example, an external display device.

[0103] After executing the composite 3D image output process (step ST6190), the image output device 10F proceeds to an end determination process (step ST6200). In the termination determination process, a control unit (not shown) of the image output device 10F determines whether to terminate the processing of the image output device 10E. The control unit (not shown) determines whether to terminate the processing of the image output device 10F in accordance with, for example, an external termination command or an execution program. When the control unit (not shown) determines not to end the processing of the image output device 10F ("NO" in step ST6200), the process proceeds to step ST6160, and the process is repeated from step ST6160. When the control unit (not shown) determines to end the process of the image output device 10F ("YES" in step ST6200), the image output device 10F ends the process.

[0104] In this embodiment, the following configuration is further disclosed. In the case where the combination of the lens group, the imaging unit, and the three-dimensional image constructing unit is three or more sets, The angle between the optical axes of the two lens groups is: the angle is an angle between the optical axis of the lens group in one of the three or more combinations as a reference and the optical axis of the lens group in the other combinations, 1. An image output device comprising: As a result, the present disclosure has an effect of providing an image output device that can further improve the resolution in the depth direction with a minimum configuration. Furthermore, when the configuration according to the present disclosure is applied to the image output method, the same effect as the above effect is achieved.

[0105] Here, a hardware configuration for realizing the functions of the present disclosure will be described. FIG. 24 is a diagram illustrating a first example of a hardware configuration for realizing the functions according to the configuration of the present disclosure. FIG. 25 is a diagram illustrating a second example of a hardware configuration for realizing the functions according to the configuration of the present disclosure. Part of the image output systems 1A, 1E, 1F and the whole or part of the image output devices 10, 10A, 10E, 10F of the present disclosure are realized by hardware such as that shown in FIG. 24 or FIG. 25, respectively.

[0106] Each of parts of the image output systems 1A, 1E, 1F and all or part of the image output devices 10, 10A, 10E, 10F is configured, for example, with a processor 10001, a memory 10002, an input / output interface 10003, and a communication circuit 10004, as shown in FIG. The processor 10001 and the memory 10002 are installed in, for example, a computer. The memory 10002 stores the computer, three-dimensional image reconstruction units 100, 100A, 100E, 100F, three-dimensional image acquisition units 110 (110-1, 110-2, . . . , 110-n (n ≧ 2)), 110A (110A-1, 110A-2, . . . , 110A-n (n ≧ 2)), 110E, 110F, angle acquisition units 120 (120-1, 120-2, . . . , 120-n (n ≧ 2)), 120A (120A-1, 120A-2, . . . , 120A-n (n ≧ 2)), 120E, 120F, combination processing units 130, 130A, 130E, 130F, three-dimensional image construction units A program for causing the imaging device 100 to function as at least one of the imaging device 230 (230-1, 230-2, . . . , 230-n (n ≧ 2)), the imaging synchronization unit 300, the angle control unit 500 (500-1, 500-2, . . . , 500-n (n ≧ 2)), the processing order command unit 600, or a control unit not shown is stored. The processor 10001 reads out and executes the program stored in the memory 10002, whereby the three-dimensional image reconstruction units 100, 100A, 100E, 100F, the three-dimensional image acquisition units 110 (110-1, 110-2, . . . , 110-n (n ≧ 2)), 110A (110A-1, 110A-2, . . . , 110A-n (n ≧ 2)), 110E, 110F, the angle acquisition units 120 (120-1, 120-2, . . . , 120-n (n ≧ 2)), 120A (120A-1, 120A-2, . . . , 120A-n (n ≧ 2)), 120E, 120F, the combination processing units 130, 130A, 130E, 130F, the three-dimensional image construction unit At least one function of the imaging unit 230 (230-1, 230-2, . . . , 230-n (n ≧ 2)), the imaging synchronization unit 300, the angle control unit 500 (500-1, 500-2, . . . , 500-n (n ≧ 2)), the processing order command unit 600, or a control unit not shown is realized. Furthermore, a storage unit (not shown) is realized by the memory 10002 or another memory (not shown). Furthermore, the communication circuit 10004 realizes a communication unit (not shown).

[0107] The processor 10001 may be, for example, a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor, a microcontroller, or a digital signal processor (DSP). The memory 10002 may be a RAM (Random Access Memory), a ROM (Read Only Memory), an EPROM (Erasable Programmable ROM), or an EEPROM (Electrically Erasable The storage device may be a non-volatile or volatile semiconductor memory such as a Programmable Read Only Memory (PRAM) or a flash memory, a magnetic disk such as a hard disk or a flexible disk, an optical disk such as a CD (Compact Disc) or a DVD (Digital Versatile Disc), or a magneto-optical disk. The processor 10001 and the memory 10002 or the communication circuit 10004 are connected in a state in which they can transmit data to each other. The processor 10001, the memory 10002, and the communication circuit 10004 are also connected via the input / output interface 10003 in a state in which they can transmit data to other hardware.

[0108] Or, in a part of the image output system 1A, 1E, 1F and in the whole or part of the image output device 10, 10A, 10E, 10F, the three-dimensional image reconstruction unit 100, 100A, 100E, 100F, the three-dimensional image acquisition unit 110 (110-1, 110-2, . . . , 110-n (n ≧ 2)), 110A (110A-1, 110A-2, . . . , 110A-n (n ≧ 2)), 110E, 110F, the angle acquisition unit 120 (120-1, 120-2, . . . , 120-n (n ≧ 2)), 120A (120A-1, 120A-2, . . . , 120A-n (n ≧ 2)), 120E, 120F, and the combination processing unit At least one function of 130, 130A, 130E, 130F, the three-dimensional image construction unit 230 (230-1, 230-2, ..., 230-n (n >= 2)), the shooting synchronization unit 300, the angle control unit 500 (500-1, 500-2, ..., 500-n (n >= 2)), the processing order command unit 600, or a control unit (not shown) may be realized by a dedicated processing circuit 20001 as shown in FIG. 25.

[0109] The processing circuit 20001 may be, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), an FPGA (Field-Programmable Gate Array), a SoC (System-on-a-Chip), or a system LSI. (Large-Scale Integration), etc. Furthermore, the memory 20002 or another memory (not shown) implements a storage unit (not shown). The memory 20002 may be a RAM (Random Access Memory), a ROM (Read Only Memory), an EPROM (Erasable Programmable ROM), or an EEPROM (Electrically Erasable The storage device may be a non-volatile or volatile semiconductor memory such as a Programmable Read Only Memory (PRAM) or a flash memory, a magnetic disk such as a hard disk or a flexible disk, an optical disk such as a CD (Compact Disc) or a DVD (Digital Versatile Disc), or a magneto-optical disk. Furthermore, the communication circuit 20004 realizes a communication unit (not shown). The processing circuit 20001 and the memory 20002 or the communication circuit 20004 are connected in a state in which they can transmit data to each other. In addition, the processing circuit 20001, the memory 20002, and the communication circuit 20004 are connected in a state in which they can transmit data to other hardware via the input / output interface 20003. In addition, in a part of the image output system 1A, 1E, 1F and the whole or part of the image output device 10, 10A, 10E, 10F, the three-dimensional image reconstruction unit 100, 100A, 100E, 100F, the three-dimensional image acquisition unit 110 (110-1, 110-2, . . . , 110-n (n ≧ 2)), 110A (110A-1, 110A-2, . . . , 110A-n (n ≧ 2)), 110E, 110F, the angle acquisition unit 120 (120-1, 120-2, . . . , 120-n (n ≧ 2)), 120A (120A-1, 120A-2, . . . , 120A-n (n ≧ 2)), 120E, 120F, and the combination processing unit At least one function of 130, 130A, 130E, 130F, the three-dimensional image construction unit 230 (230-1, 230-2,..., 230-n (n>=2)), the shooting synchronization unit 300, the angle control unit 500 (500-1, 500-2,..., 500-n (n>=2)), the processing order command unit 600, or a control unit not shown may be realized by a separate processing circuit, or may be realized collectively by a processing circuit.

[0110] Or, in a part of the image output system 1A, 1E, 1F and in the whole or part of the image output device 10, 10A, 10E, 10F, the three-dimensional image reconstruction unit 100, 100A, 100E, 100F, the three-dimensional image acquisition unit 110 (110-1, 110-2, . . . , 110-n (n ≧ 2)), 110A (110A-1, 110A-2, . . . , 110A-n (n ≧ 2)), 110E, 110F, the angle acquisition unit 120 (120-1, 120-2, . . . , 120-n (n ≧ 2)), 120A (120A-1, 120A-2, . . . , 120A-n (n ≧ 2)), 120E, 120F, and the combination processing unit Some of the functions of 130, 130A, 130E, 130F, three-dimensional image construction unit 230 (230-1, 230-2, ..., 230-n (n >= 2)), shooting synchronization unit 300, angle control unit 500 (500-1, 500-2, ..., 500-n (n >= 2)), processing order command unit 600, or a control unit not shown in the figure may be realized by a processor 10001 and memory 10002, and the remaining functions may be realized by a processing circuit 20001.

[0111] It should be noted that, within the scope of this disclosure, the embodiments may be freely combined, any component of each embodiment may be modified, or any component of each embodiment may be omitted. [Industrial Applicability]

[0112] The present disclosure can improve the depth resolution of three-dimensional images in light field imaging, and is therefore suitable for use in three-dimensional image output devices, three-dimensional image output systems, and three-dimensional image output programs that utilize light field imaging technology. [Explanation of symbols]

[0113] 1A, 1E, 1F Image output system, 10, 10A, 10E, 10F Image output device, 100, 100A, 100E, 100F Three-dimensional image reconstruction unit, 110 (110-1, 110-2, . . . , 110-n (n ≧ 2)), 110A (110A-1, 110A-2, . . . , 110A-n (n ≧ 2)), 110E, 110F Three-dimensional image acquisition unit, 120 (120-1, 120-2, . . . , 120-n (n ≧ 2)), 120A (120A-1, 120A-2, . . . , 120A-n (n ≧ 2)), 120E, 120F Angle acquisition unit, 130, 130A, 130E, 130F Combination processing unit, 200 (200-1, 200-2, . . . , 200-n (n ≧ 2)) LFIS (light field imaging system), 210 (210-1, 210-2, . . . , 210-n (n ≧ 2)) Lens group, 211 (211-1, 211-2, . . . , 211-n (n ≧ 2)) Main lens unit, 212 (212-1, 212-2, . . . , 212-n (n ≧ 2)) Array lens unit, 220 (220-1, 220-2, . . . , 220-n (n ≧ 2)) Imaging unit, 230 (230-1, 230-2, . . . , 230-n (n ≧ 2)) 3D image constructing unit, 250 (250-1, 250-2, . . . , 250-n (n ≧ 2)) optical axis, 300 imaging synchronization unit, 400 light source, 500 (500-1, 500-2, . . . , 500-n (n ≧ 2)) angle control unit, 510 (510-1, 510-2, . . . , 510-n (n ≧ 2)) angle designation mechanism, 600 processing order command unit, 1000 3D space, 1010 (1010-1, 1010-2, . . . , 1010-n (n ≧ 2)) Focal plane of the main lens, 1020 (1021 (1021-1, 1021-2, , 1021-n (n ≧ 2)), 1022 (1022-1, 1022-2, , 1022-n (n ≧ 2)), 1023 (1023-1, 1023-2, , 1023-n (n ≧ 2)), 1024 (1024-1, 1024-2, , ·,1024-n(n≧2)),1025(1025-1,1025-2,···,1025-n(n≧2)),1026(1026-1,1026-2,··· ,1026-n(n≧2)),1027(1027-1,1027-2,...,1027-n(n≧2)),1028(1028-1,1028-2,...,1028-n (n≧2)),···) refocus plane, 1030 (1034, 1035) voxel (three-dimensional pixel), 1100 imaging object, 1110 conventional three-dimensional image, 10001 processor, 10002 memory, 10003 input / output interface, 10004 communication circuit, 20001 processing circuit, 20002 memory, 20003 input / output interface, 20004 communication circuit. ,

Claims

1. An image output device that outputs a three-dimensional image using an image captured through a lens group having a main lens and a lens array, a three-dimensional image reconstruction unit that outputs a composite three-dimensional image using each image captured through the two lens groups, the optical axes of which intersect with each other, and an angle between the optical axes of the two lens groups; The angle between the optical axes of the two lens groups is: the angle is obtained from an angle designation mechanism that changes the direction of the optical axis of the main lens in each of the lens groups by changing the angle of the lens group to a designated angle, 1. An image output device comprising:

2. The three-dimensional image reconstruction unit includes: a three-dimensional image acquisition unit that acquires, for each of the lens groups, a three-dimensional image constructed using images captured via the lens groups; an angle acquisition unit that acquires an angle between the optical axes of the two lens groups; a combination processing unit that uses the three-dimensional images acquired by the three-dimensional image acquisition unit and the angles acquired by the angle acquisition unit to output the composite three-dimensional image, which is an image obtained by combining the three-dimensional images; Equipped with 2. The image output device according to claim 1.

3. a plurality of the lens groups each including the main lens and the lens array; a plurality of imaging units configured to capture an image through each of the lens groups and output imaging data; A plurality of three-dimensional image constructing units each constructing a three-dimensional image using images captured by the imaging unit; Further equipped with The three-dimensional image acquisition unit acquires the three-dimensional image constructed by the three-dimensional image construction unit.

3. The image output device according to claim 2.

4. The three-dimensional image constructing unit constructs a three-dimensional image by performing a refocusing process from a focal position of the main lens toward the main lens.

4. The image output device according to claim 3.

5. The three-dimensional image constructing unit constructs a three-dimensional image by performing a refocusing process from a focal position of the main lens in a direction opposite to the direction of the main lens.

4. The image output device according to claim 3.

6. wherein the two sets of lens groups are a first lens group and a second lens group; a distance between a position of the main lens in the first lens group and a focal position of the main lens is shorter than a distance between a position of the main lens in the second lens group and a focal position of the main lens, The three-dimensional image constructing unit includes: Among the plurality of three-dimensional image constructing units, The at least one three-dimensional image constructing unit, a first three-dimensional image constructing unit, A refocusing process is performed from a focal position of a main lens in the first lens group toward the main lens to form a three-dimensional image; A second three-dimensional image constructing unit which is at least one three-dimensional image constructing unit different from the first three-dimensional image constructing unit, performing a refocus process from a focal position of a main lens in the second lens group in a direction opposite to the direction of the main lens to form a three-dimensional image; 4. The image output device according to claim 3.

7. In the case where the combination of the lens group, the imaging unit, and the three-dimensional image constructing unit is three or more sets, The angle between the optical axes of the two lens groups is: the angle is an angle between the optical axis of the lens group in one of the three or more combinations as a reference and the optical axis of the lens group in the other combinations, 4. The image output device according to claim 3.

8. 1. An image output method by an image output device that outputs a three-dimensional image using an image captured through a lens group having a main lens and a lens array, comprising: a three-dimensional image reconstructing step in which a three-dimensional image reconstructing unit of the image output device outputs a composite three-dimensional image using each image captured through each of the two sets of lens groups whose optical axes intersect with each other, and the angle formed by the optical axes of the two sets of lens groups; Equipped with The angle between the optical axes of the two lens groups used in the three-dimensional image reconstruction step is an angle obtained from an angle designation mechanism that changes the direction of the optical axis of the main lens in each of the lens groups by changing the angle of the lens group to a designated angle, 13. An image output method comprising: