Image display device

Abstract

To display the image of a subject having a three-dimensional shape in the air as a stereoscopic image.SOLUTION: An image display device comprises: an input unit to which are inputted a plurality of image signals outputted from a plurality of imaging pixels that are arranged in correspondence to a microlens for imaging and that receive light having passed through the microlens for imaging; a plurality of arrayed microlenses for display; a plurality of display pixels that are arranged for each microlens for display and that emit light to the microlenses for display; a generation unit that generates image data for display on the basis of a plurality of image signals inputted to the input unit; and a control unit that causes the display pixels to emit light on the basis of the image data for display. When the incident light array pattern of imaging pixels corresponding to the microlens for imaging and the emission array pattern of the display pixels corresponding to the microlenses for display are not the same, the generation unit allocates image signals to display pixels where imaging pixels are arranged at positions that are point symmetrical around the pseudo-optical axis of the microlenses.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an image display device. [Background technology]

[0002] BACKGROUND ART Display devices that display stereoscopic images by integral photography have been known (for example, see Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2008-216340 Summary of the Invention [Problem to be solved by the invention]

[0004] However, in conventional display devices, an image captured as a three-dimensional image is projected and displayed, and therefore there is a problem in that a gradient index lens or the like must be used to display the image as an erect image. [Means for solving the problem]

[0005] The image display device according to the invention of claim 1 comprises an input section that receives a plurality of image signals output from a plurality of imaging pixels that are arranged corresponding to the imaging microlenses and receive light that has passed through the imaging microlenses, an array of display microlenses, a plurality of display pixels that are arranged for each display microlens and emit light to the display microlenses, a generation section that generates display image data based on the plurality of image signals input to the input section, and a control section that causes the display pixels to emit light based on the display image data, and when the arrangement pattern of the incident light of the imaging pixels corresponding to the imaging microlenses is not the same as the arrangement pattern of the emission of the display pixels corresponding to the display microlenses, the generation section assigns the image signal to the display pixels that are arranged at positions that are point-symmetrical with respect to the pseudo-optical axis of the microlens. [Effects of the Invention]

[0006] According to the present invention, an image of a subject having a three-dimensional shape can be displayed as a three-dimensional image in the air. [Brief explanation of the drawings]

[0007] [Figure 1] 1 is a diagram illustrating a configuration of a digital camera according to an embodiment of the present invention. [Figure 2] FIG. 2 is a diagram illustrating an example of an arrangement of microlenses and an imaging element. [Figure 3] 10A and 10B are diagrams illustrating the positional relationship between a microlens and a cardinal pixel according to an embodiment. [Figure 4] FIG. 2 is a diagram illustrating the principle of generating a composite image. [Figure 5] FIG. 10 is a diagram showing the relationship between an integration area for generating a composite image and imaging pixels. [Figure 6] 10A and 10B are diagrams illustrating an example of the positional relationship of imaging pixels that output image signals to be integrated into a base signal. [Figure 7] FIG. 10 is a diagram illustrating an example of the relationship between annular zones and microlenses. [Figure 8]1 is a diagram illustrating a configuration of a display device according to an embodiment. [Figure 9] 10A and 10B are diagrams illustrating an example of display pixels to which image signals output from imaging pixels are assigned. [Figure 10] 10A and 10B are diagrams illustrating a light cross section of a light beam emitted from a light point and cut off at a light receiving surface of an imaging pixel. [Figure 11] 10A and 10B are diagrams illustrating the relationship between a microlens and a cross section of light. [Figure 12] 10A and 10B are diagrams illustrating the relationship between a microlens and a cross section of light. [Figure 13] 10A and 10B are diagrams illustrating a cross section of light when the area division is expanded to a cardinal microlens. [Figure 14] 10A and 10B are diagrams illustrating divided regions when a light spot is decentered with respect to the pseudo optical axis of a cardinal microlens. [Figure 15] FIG. 10 is a diagram illustrating the depth of a displayed aerial image. [Figure 16] FIG. 2 is a diagram showing the relationship between the focal position and the surface of a display pixel. DETAILED DESCRIPTION OF THE INVENTION

[0008] The digital camera of this embodiment is configured to be able to generate image data with an arbitrary focal position set. When a subject having a three-dimensional shape is photographed with this digital camera, the generated image data contains information about the three-dimensional shape (three-dimensional information). The digital camera of this embodiment displays an image corresponding to the image data containing such three-dimensional information as a three-dimensional stereoscopic image (aerial image) that can be viewed by a user. This will be described in detail below.

[0009] FIG. 1 shows the configuration of a digital camera according to an embodiment. Digital camera 1 is configured so that an interchangeable lens 2 having a photographing lens L1 can be attached or detached. Digital camera 1 includes an imaging unit 100, a control circuit 101, an A / D conversion circuit 102, a memory 103, an operation unit 108, a memory card interface 109, and a display device 110. Imaging unit 100 includes a microlens array 12 in which a large number of microlenses 120 are arranged two-dimensionally, and an image sensor 13. In the following description, the z-axis is set parallel to the optical axis of photographing lens L1, and the x-axis and y-axis are set perpendicular to each other in a plane perpendicular to the z-axis.

[0010] The photographing lens L1 is composed of a group of multiple optical lenses and forms an image of a light beam from a subject near its focal plane. For ease of explanation, the photographing lens L1 is represented by a single lens in FIG. 1. Behind the photographing lens L1, a microlens array 12 and an image sensor 13 are arranged in this order, two-dimensionally, in a plane perpendicular to the optical axis. The image sensor 13 is composed of a CCD or CMOS image sensor with multiple photoelectric conversion elements. The image sensor 13 captures the subject image formed on the imaging surface and, under the control of a control circuit 101, outputs a photoelectric conversion signal (image signal) corresponding to the subject image to an A / D conversion circuit 102. Details of the imaging unit 100 will be described later.

[0011] The A / D conversion circuit 102 performs analog processing on the image signal output by the image sensor 13 and then converts it into a digital image signal. The control circuit 101 is composed of a CPU, memory, and other peripheral circuits. Based on a control program, the control circuit 101 performs predetermined calculations using signals input from each component of the digital camera 1, and sends control signals to each component of the digital camera 1 to control the photographing operation. The control circuit 101 also determines the aperture value of the composite image selected by the user based on an operation signal input from an operation unit 108 in response to operation of an aperture value input button, as will be described later.

[0012] The control circuit 101 functionally includes an image integration unit 105, an image pattern generation unit 106, and a display control unit 107. The image integration unit 105 generates composite image data from an image signal using a composite pixel affiliation table corresponding to the aperture value of the composite image determined in response to operation of the aperture value input button. The image pattern generation unit 106 creates display image data from the composite image data generated by the image integration unit 105, as described below, to display an aerial image having three-dimensional information on a display device 110, as described below. The display control unit 107 controls driving of the display device 110, outputs the display image data generated by the image pattern generation unit 106 to the display device 110, and causes the display device 110 to display the corresponding aerial image having three-dimensional information. The image integration unit 105 and the image pattern generation unit 106 will be described in detail below.

[0013] The memory 103 is a volatile storage medium used to temporarily store image signals digitally converted by the A / D conversion circuit 102, and data during or after image processing, image compression processing, and display image data creation processing. The memory card interface 109 is an interface to which a memory card 109a can be attached / detached. The memory card interface 109 is an interface circuit that writes image data to the memory card 109a and reads image data recorded on the memory card 109a under the control of the control circuit 101. The memory card 109a is a semiconductor memory card such as a CompactFlash (registered trademark) or an SD card.

[0014] The operation unit 108 accepts user operations and outputs various operation signals according to the operation content to the control circuit 101. The operation unit 108 includes an aperture value input button, a power button, a release button, other setting menu display switching buttons, a setting menu confirmation button, etc. The aperture value input button is operated by the user when inputting the aperture value F of the composite image. When the user operates the aperture value input button to select the aperture value F, the operation unit 108 outputs an operation signal to the control circuit 101.

[0015] In response to a command from the control circuit 101, the display device 110 displays display data created by the control circuit 101 based on image data recorded on the memory card 109a in playback mode. The display device 110 also displays a menu screen for setting various operations of the digital camera 1. Details of the display device 110 will be described later.

[0016] Next, the configuration of the imaging unit 100 will be described in detail. As described above, the imaging unit 100 has the microlens array 12 and the imaging element 13. The microlens array 12 is composed of a plurality of microlenses 120 arranged two-dimensionally. The imaging element 13 has a pixel array 130 that receives light that has passed through each of the microlenses 120, and is arranged in an arrangement pattern corresponding to the microlenses 120. Each pixel array 130 is composed of a plurality of photoelectric conversion elements 131 (hereinafter referred to as imaging pixels 131) arranged two-dimensionally.

[0017] FIG. 2(a) shows a plan view of the microlenses 120 arranged in the microlens array 12 in the XY plane. As shown in FIG. 2(a), the microlens array 12 has a plurality of microlenses 120, each formed, for example, in a hexagonal shape, arranged in a honeycomb pattern on the XY plane. Note that FIG. 2(a) shows some of the microlenses 120 arranged in the microlens array 12. FIG. 2(b) is a diagram illustrating the positional relationship between the microlens array 12 and the image sensor 13 in the optical axis direction (z-axis direction) of the imaging lens L1. As shown in FIG. 2(b), the image sensor 13 is disposed at a position spaced apart by the focal length f of the microlenses 120. That is, a pixel array 130 having a plurality of imaging pixels 131 is disposed at a position spaced apart by the focal length f of the microlenses 120 corresponding to each pixel array 130. 2(b) shows a plurality of microlenses 120 provided in the microlens array 12, a plurality of pixel arrays 130 provided in the imaging element 13, and a portion of a plurality of imaging pixels 131.

[0018] The image integrator 105 generates composite image data using the image signal output from the imaging element 13 having the above-described configuration. The image integrator 105 combines an image signal (hereinafter referred to as a cardinal signal) output from a specific imaging pixel 131 (hereinafter referred to as a cardinal pixel 132 (FIG. 3)) among the imaging pixels 131 included in the pixel array 130 provided corresponding to a certain microlens 120, with image signals output from imaging pixels 131 included in the pixel array 130 corresponding to the microlens 120 corresponding to the cardinal pixel 132 and other microlenses 120 provided nearby. As a result, the image integrator 105 generates a composite image signal equivalent to one pixel. The image integrator 105 performs the above process on all cardinal pixels corresponding to each microlens 120, and adds up the generated composite image signals to generate composite image data.

[0019] As described above, the image integration unit 105 refers to the integration pixel affiliation table when generating a composite image signal. The integration pixel affiliation table indicates which microlens 120 corresponds to which imaging pixel 131 that outputs an image signal to be integrated into a base signal, and at what position in the pixel array 130. The process by which the image integration unit 105 generates a composite image signal using the image signals output from the imaging pixels 131 will be described below.

[0020] FIG. 3 shows each microlens 120, i.e., a cardinal pixel 132 provided corresponding to each pixel array 130. FIG. 3 shows some of the multiple microlenses 120, and represents the cardinal pixel 132 among the multiple imaging pixels 131. In FIG. 3, the cardinal pixel 132 is arranged corresponding to the pseudo-optical axis of the microlens 120. In this embodiment, the pseudo-optical axis is described as the intersection between the center of a light beam incident from the pupil of the imaging lens L1 and the main surface of the microlens 120. FIG. 3 shows a case where the geometric center of the microlens 120 and the pseudo-optical axis coincide with each other. In the following description, the microlens 120 corresponding to the cardinal pixel 132 will be referred to as the cardinal microlens 121.

[0021] -Generation of synthetic image signals- First, the principle of generating a composite image will be described when the image of the subject shown in FIG. 4(a) is formed at the vertex of the microlens 120, i.e., when the focal plane S is at the vertex of the microlens 120. In this case, light beams r1 to r7 from the subject are incident on imaging pixels 131 of a pixel array 130 provided corresponding to the microlens 120. The image integrator 105 generates a composite image signal corresponding to one pixel of composite image data by integrating image signals output from the imaging pixels 131 indicated by diagonal lines among the imaging pixels 131 shown in FIG. 4(a). The image integrator 106 performs this process for the pixel array 130 corresponding to all microlenses 120, thereby generating composite image data.

[0022] Next, the principle of generating a composite image signal for an image of a subject formed on a certain focal plane (imaging plane) will be described. As shown in FIG. 4(b), when the focal plane S is located away from the vertices of the microlenses 120, the light beams r1 to r5 from the subject are incident on different microlenses 120. Therefore, in order to generate a composite image signal, the image integration unit 105 must also use image signals from the imaging pixels 131 arranged corresponding to the microlenses 120 arranged near the cardinal microlens 121. Note that FIG. 4(b) shows the chief rays of the light beams r1 to r5 as representatives.

[0023] The image integration unit 105 generates a composite image signal corresponding to one pixel in the composite image data (imaging area of ​​the composite image) by integrating all image signals output from the imaging pixels 131 included in an integration area determined in accordance with the aperture value F of the composite image. The integration area is represented by a circle with a diameter D. The diameter D of the integration area is expressed by the following equation (1) using the aperture value F (aperture value of the composite image data) determined in accordance with the operation of the aperture value input button 108a and the focal length f of the microlens 120. D=f / F (1)

[0024] FIG. 5 shows the relationship between the integration area Rs and the imaging pixels 131. As described above, the image integration unit 106 integrates the image signals output from all imaging pixels 131 covered by the integration area Rs, which is represented as a circular area. In FIG. 5, the imaging pixels 131 that output the integrated image signals are indicated by diagonal lines. Because the microlens 120 is one of the lenses that make up the microlens array 12, the integration area Rs cannot be made larger than the diameter of each microlens 120 allowed by the arrangement of the microlenses 120. Therefore, the maximum aperture value Fmax allowed in the composite image data is expressed by the following equation (2). Note that in equation (2), "s" represents the length of one side of the imaging pixel 131. The minimum aperture value Fmin in the composite image data is the F-number of the microlens 120. Fmax=f / s (2)

[0025] The composite image signal obtained by integrating the image signals output from the pixel array 130 including the cardinal pixel 132 by the image integration unit 105, i.e., the integrated value, is expressed by the following equation (3). In equation (3), P represents the output value of the image signal output from the imaging pixel 131. Furthermore, "i" in equation (3) represents the imaging pixel 131 covered by the integration region Rs when the aperture value of the composite image is F, and "0" represents the microlens 120 arranged corresponding to the pixel array 130 including the cardinal pixel 132, i.e., the cardinal microlens 121.

number

[0026] As described above, the image integrating unit 105 also performs integration using image signals output from imaging pixels 131 included in the pixel array 130 corresponding to the microlenses 120 provided near the cardinal point microlens 121. That is, the image integrating unit 105 integrates output values ​​of pixel signals from all imaging pixels 131 included in the set F{i} of imaging pixels 131 covered by the integration region Rs determined by the aperture value F of the composite image, and arranged corresponding to the nearby microlenses 120 including the cardinal point microlens 121. In this case, the output value P is expressed by the following equation (4). Note that "t" in equation (4) represents the nearby microlenses 120 including the cardinal point microlens 121.

number

[0027] Fig. 6 shows the relationship between the imaging pixel 131 that outputs the image signal used when generating one composite image signal by the image integration unit 105, and the cardinal point microlens 121 and the nearby adjacent microlenses 120a-120f. Note that Fig. 6 does not show the imaging pixel 131 that does not output the image signal used to generate the composite image signal. When the imaging pixels 131 distributed among the cardinal point microlens 121 and the adjacent microlenses 120a-120f shown in Fig. 6 are collected together, they constitute a region defined by the aperture value F of the composite image shown in Fig. 5, i.e., a plurality of imaging pixels 131 covered by the integration region Rs.

[0028] When the image integrator 105 performs the above-described processing to integrate the image signal, it is important to know which microlens 120 the imaging pixel 131 that outputs the image signal to be added to the base signal is located at and at which position in the pixel array 130. Therefore, a table indicating which microlens 120a-120f the imaging pixel 131 indicated by "i" in equations (3) and (4) corresponds to, i.e., the distribution of the imaging pixels 131, is stored in a predetermined storage area as a composite pixel affiliation table. The image integrator 105 then references this composite pixel affiliation table when generating a composite image signal. The composite pixel affiliation table is expressed by the following equation (5). t=T d (i) (5)

[0029] The principle of creating the composite pixel affiliation table will be explained below. FIG. 10 shows a light cross section LFD of a light beam LF emitted from a light point LP in the microlens array 12 and cut off at the light-receiving surface of an imaging pixel 131. As shown in FIG. 10, the light beam LF spreading from the light point LP is limited in its spreading angle by the imaging lens L1 in the preceding stage. Therefore, the light beam LF incident on each microlens 120 does not extend beyond the area covered by the microlens 120 (in FIG. 10, the light cross sections LFDc and LFDe are drawn as if they partially extend beyond the covered area). This can also be explained by the fact that the light-receiving surface of the imaging pixel 131 is optically conjugate with the pupil of the imaging lens L1. When capturing an image through the imaging lens L1, a capturing pupil image, i.e., a light boundary, is formed within the area covered by the microlens 120, and the light beam LF does not enter outside of this boundary.

[0030] The following explanation is based on the above points. In the microlens array 12 shown in FIG. 10, by integrating the amounts of light incident on the imaging pixels 131a to 131e corresponding to the light cross sections LFDa to LFDe (collectively referred to as LFD) of the light beam LF, the total radiation amount of the light beam LF from the light point LP that is limited to the pupil of the photographing lens L1 can be obtained. Therefore, when integrating image signals, the image integrator 105 simply calculates the light cross section LFD of the light receiving element surface of the imaging pixel 131 relative to the coordinate of the light point LP in the z-axis direction. Conversely, if display elements are provided and light is emitted from each display element corresponding to each light cross section LFD of the light beam LF, there will always be a light beam LF traveling in the same direction as the incident light, and the "light point LP" will be the point of accumulation of the light beam LF.

[0031] As described above, the angle of the light beam LF spreading from the light point LP is determined by the pupil of the imaging lens L1, i.e., the F-number of the imaging lens L1. In cases where the imaging lens L1 is not present, such as in a display system, the maximum aperture (minimum F-number) is determined by the F-number of the microlens 120. Therefore, the aperture can be limited by using only the center of the coverage area of ​​the microlens 120.

[0032] Using FIG. 11, how many or which microlenses 120 correspond to which light cross section LFD will be explained by projecting the spread of the light beam LF from the light point LP onto the microlenses 120. For convenience of explanation, FIG. 11 shows the case where the microlenses 120 are arranged in a square array. Also, in FIG. 11, the light beam LF spreading from the light point LP is shown when the position of the light point LP in the z-axis direction is the focal length f of the microlenses 120 and when it is twice that, 2f. In FIG. 11, the spread of the light beam LF when the position of the light point LP is f is shown by a broken line, and when it is 2f is shown by a chain line. When the light point LP is at the position of the focal length f of the microlenses 120, the spread of the light beam LP is determined by the microlenses 120 (the light cross section LFD is a circle, but the microlenses 120 are optically effective up to the edge of the square). Since the square shape is obtained by dividing the microlens 120, the light beam LF is incident on one microlens 120. As a result, the microlens 120 corresponding to one light spot LP is determined.

[0033] When the position of the light point LP is at the focal length f of the microlens 120, the light beam LF spreads as light from a circular aperture over the entire area directly below the microlens 120. Therefore, it is sufficient to select image signals from all of the imaging pixels 131 contained within the circle inscribed in the square area. When the absolute value of the position of the light point LP is smaller than the focal length f, the light beam LF spreads without converging within the area directly below the microlens 120. However, due to the restriction on the spread angle of the incident light beam LF, the light cross section LFD remains within the covered area.

[0034] Here, we will explain the case where the position of the light spot LP is at 2f. Figure 12 shows the microlenses 120 involved in this case. As shown in Figure 12(a), the relevant microlenses 120 are themselves, that is, the base microlens 121 and the eight microlenses 120 adjacent to it. When considering the aperture restriction by the microlenses 120, the light cross section LFD exists within the covered area shown by diagonal lines in Figure 12(a). In this case, the light cross section LFD by each microlens 120 is the area shown by diagonal lines in Figure 12(b).

[0035] 12(b), the coverage area of ​​one base point microlens 121 is divided and allocated to adjacent microlenses 120. The total area obtained by accumulating the divided and allocated coverage areas (partial areas) is the aperture area of ​​one microlens 120. Therefore, the size of the total area of ​​the light cross section LFD is the same regardless of the position of the light point LP, so when calculating the total area by accumulating the partial areas, it is sufficient to determine the microlens 120 to which each partial area belongs.

[0036] 11 shows the relationship between the position of the light spot LP and the magnification, i.e., the number of microlenses 120 adjacent to the base microlens 121, and this is applied to a virtual aperture area. In this embodiment, the aperture area is divided by an array of microlenses 120 reduced by the magnification, and fragments of the aperture area are arranged at the same positions within the microlenses 120 defined by this. An example will be described in which a square circumscribing the aperture area is reduced by a magnification of 2, and the aperture area is divided (area division) by the array of microlenses 120.

[0037] FIG. 13 shows the light cross section LFD when the above-mentioned area division is expanded to the base point microlens 121. If a similar area division is performed according to the magnification, the pattern of the light cross section LFD for the magnification, that is, the light point LP, is obtained. Specifically, when the diameter of the microlens 120 (the size of one side of the microlens) is g, the opening area is divided by a grid with a width of g / m. Magnification is expressed as m=y / f, the ratio of the height (position) y of the light spot LP to the focal length f of the microlens. The ratio m can also have a negative sign. When the sign of the ratio m is negative, it is assumed that the light spot LP is located closer to the image sensor 13 than the microlens 120.

[0038] In the above example, the light spot LP was described as existing on a pseudo-optical axis, which is the lens center axis of a certain microlens 120. However, even if the light spot LP is actually decentered, there is no problem in the calculation. If calculations could only be performed on the lens center, the two-dimensional resolution of the composite image would be equal to the number of microlenses 120, but this is usually completely insufficient. The reason for this is that if the number of imaging pixels 131 covered by the microlens 120 is 100, the resolution of the composite image would be 1 / 100 of the number of pixels. Therefore, to obtain a composite image with 1 million pixels, 100 million imaging pixels 131 are required. Therefore, synthesis is performed at an eccentric position so that multiple light spots LP can be accommodated within the microlens 120.

[0039] Since the product of the area covered by the microlenses 120 and the number of microlenses 120 is approximately equal to the total number of imaging pixels 131, combining images using each of a plurality of decentered points within one microlens 120 as a base point is equivalent to using superimposed image signals from the imaging pixels 131. In other words, the light beams LF from the decentered light points LP are superimposed and present on the imaging pixel 131. However, when the magnification is 1x, this calculation is simply an interpolation process and does not substantially contribute to improving resolution. This indicates that if an image is formed near the vertex of the microlens 120, optical information in the depth direction is lost.

[0040] Fig. 14 shows divided regions for a light spot LP that is decentered to the left with respect to the pseudo-optical axis of the cardinal microlens 121. A case will be described in which the light spot LP is decentered by p to the left in Fig. 14 from the center of the cardinal microlens 121 (lens diameter is g), i.e., from the pseudo-optical axis, and the height (position) of the light spot LP is 2f. In Fig. 14, point O1 indicates the decentered light spot LP, and point O2 indicates the pseudo-optical axis. In this case, if the microlens 120 shown in Fig. 13 is shifted by p to the right in the figure and the aperture region is divided, the divided regions shown in Fig. 14 can be obtained.

[0041] If the microlens 120 is divided into 16 pieces, and the coordinates of the lens center (pseudo optical axis) are (0,0), then by taking the patterns at positions -g / 2, -g / 4, 0, g / 4, and g / 2 on the x and y axes, respectively, and accumulating the resulting divided areas and the entire area, a group of 16 light spots can be obtained for one microlens 120.

[0042] - Composite pixel affiliation table creation process - When integrating image signals, the image integration unit 105 refers to the composite pixel affiliation table. As described above, this composite pixel affiliation table identifies the positions of the imaging pixels 131 that output image signals to be integrated into the base signal in the pixel array 130 that correspond to the base microlens 121 and the microlenses 120 provided nearby.

[0043] When the focal position y of the composite image and the aperture value F (depth of field) of the composite image are determined, the image integration unit 105 creates a composite pixel affiliation table for the imaging pixels 131 that output image signals to be integrated into the base signal. As described above, the focal position of the composite image determines which imaging pixel 131 corresponds to which microlens 120 and from which image signal is to be integrated into the base signal.

[0044] Fig. 6(a) shows a case where the focal position (focal plane) y of the composite image is on the subject side relative to the microlens array 12. Fig. 6(b) shows a case where the focal position (focal plane) y of the composite image is on the imaging element 13 side relative to the microlens array 12. As shown in Figs. 6(a) and 6(b), the arrangement of the imaging pixel 131 that outputs the image signal to be integrated into the base signal varies with the focal plane position for the imaging pixel 131 corresponding to the microlens 120a. The same applies to the other microlenses 120b to 120f and the base microlens 121.

[0045] The process of creating the composite pixel affiliation table by the image integration unit 105 will be described in detail below. Assume that the focal plane of the composite image is located at a distance y from the microlens array 12, i.e., the focal length is y. Furthermore, a light beam passing through the pseudo-optical axis of the nth microlens 120 from the base microlens 121 is incident on a position at a distance x from the pseudo-optical axis of the base microlens 121, as shown in the following equation (6). Note that "d" represents the array pitch of each microlens 120. x=fnd / y (6)

[0046] Considering that the imaging pixel 131 receives a light beam focused by the corresponding microlens 120, the width l of the light from the subject at the focal position y of the composite image that is irradiated by each microlens 120 on the imaging surface of the imaging element 13 is expressed by the following equation (7). l=fd / y (7)

[0047] The width l of the light is represented by a ring-shaped area (hereinafter referred to as an annular zone) of width l on the two-dimensional plane of the image sensor 13. Therefore, in the microlens 120 located at the nth position from the base microlens 121, the light beam determined by the aperture value F of the composite image is incident on the area represented by this annular zone l. As shown in equation (7), the width of the annular zone l becomes smaller as the focal position y of the composite image becomes larger.

[0048] In this embodiment, each microlens 120 has a hexagonal shape in the xy plane as shown in FIG. 3 and is arranged in a honeycomb pattern on the microlens array 12. FIG. 7 shows annular zone l1 when n=1 and annular zone l2 when n=2 within an integration region Rs corresponding to the aperture value F of a certain composite image. As shown in FIG. 7, the annular zone l1 when n=1 is divided by the cardinal microlens 121 and microlenses 120a-120f to form partial regions Rpa-Rpg, respectively. That is, each partial region Rpa-Rpg is covered by a different microlens 120. Therefore, the image integrator 105 calculates the output value Pi,s of the image signal from the imaging pixels 131 included in each partial region Rpa-Rpg of the annular zone l1. The image integrator 105 then performs integration over the integration region Rs, i.e., over all the annular zones l, in a similar manner.

[0049] The cardinal point microlens 121 and each of the microlenses 120a to 120f basically have the same relationship with the adjacent microlenses 120. Therefore, the image integration unit 105 determines to which partial region Rp a certain imaging pixel 131 belongs, for each imaging pixel 131 included in each of the partial regions Rpa to Rpg that make up the annular zone l1.

[0050] The diameter of the integration region Rs, which includes the imaging pixels 131 that output the image signals to be integrated relative to the cardinal pixel 132, is defined as (D=f / F). The arrangement pitch d of the microlenses 120 in the x-axis direction (horizontal direction), in other words, the diameter of the circle inscribed in each hexagonal microlens 120, is defined as equal to the maximum diameter Dmax of the integration region Rs. The focal position (focal length) of the composite image is defined as y, with the virtual bending plane of the microlens 120 as the reference. In this case, the area projected onto the integration region Rs by multiplying the arrangement pitch d of each microlens 120 in the microlens array 12 by the projection magnification f / y corresponds to each of the partial regions Rp into which the annular zone l is divided by each microlens 120. Therefore, the image integration unit 105 associates the positions of the imaging pixels 131 included in the partial region Rp with the microlenses 120 corresponding to the partial region Rp, and creates the above formula (5) as a composite pixel affiliation table for the cardinal pixel 132. The position of the microlens 120 corresponding to the partial region Rp is specified as a relative position with the position of the cardinal microlens 121 as the reference.

[0051] The composite image data generated with reference to the composite pixel affiliation table described above includes three-dimensional information of a subject having a different focal position, i.e., a three-dimensional shape. Digital camera 1 according to the present embodiment generates two-dimensional display image data having three-dimensional information based on the composite image data having the three-dimensional information generated as described above, and displays a display image corresponding to the display image data on display device 100 configured to display three-dimensional images. The user then observes the three-dimensional display image as an aerial image via display device 100.

[0052] With reference to FIG. 8, a display device 110 for displaying a display image including three-dimensional information generated as described above will be described. FIG. 8 is a diagram schematically illustrating the configuration of the display device 110 in the z-axis direction. As shown in FIG. 8(a), the display device 110 includes a display 111, a display microlens array 112, and a virtual image lens 113. The display 111 is configured, for example, by a liquid crystal display or an organic EL display having a backlight, and has a plurality of display pixel arrays 114 arranged two-dimensionally. Each of the plurality of display pixel arrays 114 has a plurality of display pixels 115 arranged two-dimensionally. The display pixels 115 are controlled by the display control unit 107 described above and emit light in accordance with display image data, as will be described later.

[0053] The display microlens array 112 is composed of a plurality of display microlenses 116 arranged two-dimensionally. The display microlens array 112 is arranged on the user (observer) side in the z-axis direction, at a position spaced apart from the display pixels 115 by a focal length f' of the display microlenses 116. The display microlenses 116 are arranged in an arrangement pattern corresponding to the plurality of display pixel arrays 114. Each display microlens 116 forms an image of light emitted from the display pixels 115 in accordance with image data on a predetermined image plane on the user (observer) side in the z-axis direction.

[0054] The virtual image lens 113 is configured, for example, by a large-diameter Fresnel lens or a flat lens utilizing diffraction or the like, and is large enough to cover the entire surface of the display 111 on the xy plane. The virtual image lens 113 is disposed at a position where a user can observe the image displayed on the display 111 as a virtual image by observing the virtual image lens 113. That is, the virtual image lens 113 is disposed at a position in the z-axis direction where the image plane Q formed by the display microlenses 116 described above is inside the focal position P of the virtual image lens 113. In other words, the virtual image lens 113 is disposed so that the image plane Q is located between the virtual image lens 113 and the focal position P of the virtual image lens 113.

[0055] As shown in Fig. 8(b), the positional relationship between the display microlens array 112 and the display pixels 115 in the z-axis direction of the display device 110 described above can be considered equivalent to the positional relationship between the microlens array 112 and the imaging pixels 131 in the z-axis direction of the imaging unit 100 shown in Fig. 4. As shown in Fig. 4(b), when subject light from a certain focal position S is incident on multiple imaging pixels 131, the image pattern generation unit 106 generates display image data so that the display pixels 115 emit light in an array pattern similar to the array pattern of the incident light on the imaging pixels 131 shown in Fig. 6(a). In this case, as shown in Fig. 8(b), light beams r1 to r5 from the display pixels 115 form an image at focal position S' via the display microlenses 116.

[0056] When the correspondence between the microlenses 120 and the imaging pixels 131 expressed by Equation (5) is reproduced by the display microlenses 116 and the display pixels 115 for each pixel of the composite image data, light emitted from the display device 111 forms an image at the focal position S' corresponding to the focal position S, which differs for each pixel of the composite image data. As a result, a display image having three-dimensional information corresponding to the stereoscopic information of the composite image data is formed as an aerial image having a three-dimensional shape. In this case, the actual depth distance of the subject is reproduced in the display image in a reduced form while maintaining the sense of distance. In other words, the reciprocal of the actual distance of the subject is compressed in the aerial image. Note that FIG. 8(b) also shows the chief rays of the light beams r1 to r5 from the display pixels 115.

[0057] The image pattern generation unit 106 generates display image data corresponding to the display image described above using the image signals output from each imaging pixel 131. At this time, the image pattern generation unit 106 determines, based on the composite pixel affiliation table, the display pixel 115 to emit light at an intensity corresponding to the image signal output from a certain imaging pixel 131. In other words, the image pattern generation unit 106 assigns the image signal output from each imaging pixel 131 to the display pixel 115 arranged corresponding to the position of the imaging pixel 131. However, the direction of light from the display pixel 115 is opposite to that during image capture. Therefore, if the positional relationship between the microlens 120 and the imaging pixel 131 recorded in the composite pixel affiliation table is used as is for generating the display image data, the perspective of the observed aerial image will be reversed. For this reason, the image pattern generation unit 106 assigns the display pixel 115 to a position that is point-symmetrical with respect to the base microlens 121, i.e., an equivalent position, to the imaging pixel 131 recorded in the composite pixel affiliation table. As a result, the image plane at the time of shooting is observed as an aerial image.

[0058] First, the image pattern generation unit 106 detects the cardinal microlens 121 of the imaging unit 100 that corresponds to one of the multiple display microlenses 116. Note that data indicating the correspondence between the display microlens 116 and the cardinal microlens 121 is assumed to be stored in advance in a predetermined recording area. Then, the image pattern generation unit 106 references the composite pixel affiliation table of the detected cardinal microlens 121 to detect the imaging pixel 131 that output the image signal that forms the composite image signal and the position of that imaging pixel 131 on the imaging element 13.

[0059] After detecting the imaging pixels 131 and their positions, the image pattern generation unit 106 detects, based on the detected positions, the positions of the display pixels 115, to which the image signals from the imaging pixels 131 are assigned, relative to the display microlenses 116. The image pattern generation unit 106 then assigns the image signals output from the imaging pixels 131 to the detected display pixels 115. That is, the image pattern generation unit 106 assigns the display pixel 115a shown in FIG. 9(b) as the display pixel 115 that should emit light in response to the image signal from the imaging pixel 131a shown in FIG. 9(a). If the arrangement between the microlenses 120 and the imaging pixels 131 and the arrangement between the display microlenses 116 and the display pixels 115 cannot be considered equivalent, the image pattern generation unit 106 assigns the image signals to the display pixels 115 that are positioned normalized to the relative positions from the pseudo-optical axis of the display microlenses 116. As a result, one pixel data of the display image data is constructed based on the light beams r1 to r5 emitted from the plurality of display pixels 115.

[0060] As described above, the display device 110 displays a virtual image. Therefore, the image pattern generation unit 106 generates display image data that is inverted in the vertical direction (y-axis direction) with respect to the composite image data. That is, the image pattern generation unit 106 assigns the image signal from the imaging pixel 131 to the display pixel 115 that is arranged at a position symmetrical in the vertical direction (y-axis direction) with respect to the pseudo optical axis of the display microlens 116. For example, the image pattern generation unit 106 assigns the image signal output from the imaging pixel 131b shown in FIG. 9(a) to the display pixel 115 shown in FIG. 9(b).

[0061] The image pattern generation unit 106 performs the above processing for all display microlenses 116. If image signals output from multiple image sensors 131 are to be assigned to the same display pixel 115, the image pattern generation unit 106 superimposes and adds the multiple image signals. As a result, the image pattern generation unit 106 generates display image data. The image pattern generation unit 106 outputs the generated display image data to the display device 110 via the display control unit 107.

[0062] When each display pixel 115 emits light based on this display image data, a relief-like three-dimensional image plane is formed by the display microlenses 116 in accordance with the three-dimensional information. This three-dimensional image plane is projected to a predetermined size by the virtual image lens 113, and is observed by the user as a three-dimensional aerial image. In other words, the three-dimensional image plane formed by the display 111 and the display microlenses 116 is optically equivalent to the image plane when a three-dimensional subject is photographed with the photographing lens L1. Therefore, the image of the three-dimensional subject formed near the expected image plane is restored as a three-dimensional aerial image by the virtual image lens 113. As a result, the user can stereoscopically observe the three-dimensional aerial image on the display device 110.

[0063] According to the digital camera 1 according to the embodiment described above, the following advantageous effects can be obtained. (1) The image integration unit 105 generates composite image data having information about multiple focal positions using the image signal output from the image sensor 13, and the image pattern generation unit 106 generates display image data having three-dimensional information based on the composite image data. The display 111 has a plurality of display pixels 115 arranged two-dimensionally, and the display pixels 115 emit light beams from the plurality of display pixels in accordance with the display image data. The display microlens array 112 has a plurality of display microlenses 116 arranged two-dimensionally, which combine the light beams emitted from the plurality of display pixels 115 to form a three-dimensional image. The virtual image lens 113 is configured to enable observation of the three-dimensional image formed by the display microlenses 116. As a result, a user can observe an image of a subject having a three-dimensional shape on a screen as a three-dimensional aerial image, without using a stereoscopic display that utilizes the illusion of parallax between the right and left eyes, as in stereoscopic or lenticular methods. Therefore, since the aerial image is not based on an illusion but is actually reproduced in three dimensions, it is possible to prevent adverse physiological effects such as motion sickness and the inhibition of children's visual formation functions, which have been problems with conventional stereoscopic image displays. Furthermore, since it can be viewed without the need for special glasses, it is possible to view for long periods of time.

[0064] Furthermore, when using a hologram system, the display redundancy is 1000:1 or more, so to reproduce a 3D image with a resolution of, for example, 100,000 pixels, the display device needs to have more than 1 billion pixels. In contrast, the display device 110 of the present embodiment can display a 3D aerial image with a pixel count that is approximately 100 to 1000 times the image resolution redundancy. Furthermore, when projecting and displaying an image captured as a 3D image, a 3D aerial image can be reproduced with a simple configuration without using a refractive index gradient lens or other configuration for displaying the image as an erect image.

[0065] (2) The plurality of display microlenses 116 are arranged corresponding to the plurality of display pixels 115, and the image pattern generation unit 106 generates display image data such that one pixel of the three-dimensional image is formed by light beams emitted from the plurality of display pixels 115 arranged corresponding to the plurality of display microlenses 116. In other words, the image pattern generation unit 106 generates display image data such that the arrangement of the plurality of display pixels 115 that emit light beams is equivalent to the arrangement of the plurality of imaging pixels 131 corresponding to one pixel data of the composite image data. As a result, by assigning the image signal output from each imaging pixel 131 to the corresponding display pixel 115, display image data having three-dimensional information can be easily generated.

[0066] (3) The virtual image lens 113 is arranged so that a three-dimensional image plane is formed between the virtual image lens 113 and its focal length by the display microlenses 116. Therefore, the two-dimensional display image data generated by the image pattern generation unit 106 can be observed as a three-dimensional aerial image with a simple configuration.

[0067] The digital camera 1 according to the embodiment described above can be modified as follows. (1) Instead of displaying the display image data generated by image pattern generation unit 106 based on the composite image data generated by image integration unit 105 on display device 110, the display image data may be displayed on a monitor of an external display device (such as a personal computer or television) different from digital camera 1. In this case, the external display device reads the composite image data generated by digital camera 1 and recorded on memory card 109a. Then, using the read composite image data, the display device performs the same processing as that performed by image pattern generation unit 105 to generate display image data and output it to the monitor.

[0068] In this case, the monitor of the display device must also be configured to enable observation of a three-dimensional aerial image, similar to display device 110 of the embodiment. That is, as shown in Fig. 8, it is sufficient to have a display device in which a plurality of display pixels are arranged two-dimensionally, a microlens array in which a plurality of microlenses for forming an image of light beams from the display pixels are arranged two-dimensionally, and a virtual image lens through which the user observes the three-dimensional image formed by the microlenses as a virtual image. Furthermore, when the display device reads a multi-viewpoint image file from digital camera 1, it may use an interface such as a LAN cable or wireless communication.

[0069] (2) The display of the digital camera 1 or the external display device described above may not include the virtual image lens 113. In this case, the display image data is generated in the same manner as in the embodiment. However, the image pattern generation unit 106 does not generate the display image data so as to invert the composite image data vertically (in the y-axis direction). The image plane displayed at this time is not a reproduction of a three-dimensional image in the strict sense. It is a reproduction of the image plane compressed by the photographing lens 1 near the focal point, just as when the image was captured. Because there is a linear relationship between distance and the reciprocal of the focal length, as the distance increases, almost no three-dimensional image is reproduced. Conversely, objects that are closer to the subject appear larger even with the same depth. When an object closer than the actual subject is represented in three dimensions, considering that the eye is an optical device that unfolds on a certain image plane, the depth of the represented object should be compressed more than in reality. Therefore, based on this idea, the display device 110 represents the image plane PL1 shown by the dashed line in Fig. 15 by compressing the image plane PL1 and representing it as the image plane PL2 shown by the broken line near the lens array. If the image plane PL1 appears at a location different from the actual position, the depth may be adjusted accordingly.

[0070] The display screen is generally larger than the imaging area of ​​the image sensor 131 of the digital camera 1 that took the photograph. For example, if the image sensor size of the digital camera 1 is a so-called full size, its size is 35 x 24 (mm), which is less than 1 / 10 of the size of a normal 20-inch (40.6 x 30.5) television monitor. Even if we assume that it is 10 times larger for the sake of convenience, as will be described later, the image area of ​​the subject will be If the shooting magnification is 1 / 20 (1m position with a 50mm lens), the magnification will be Unlike 2D displays, 3D images show an object 10 times larger than the original. The displayed content is very different when displayed at twice the distance, at the same magnification and distance, or at 1 / 10 the magnification and 1 / 10 the distance. The latter results in a relatively greater sense of depth and a stronger sense of three-dimensionality.

[0071] Next, we will discuss the depth of the display. The height in the depth direction is compressed by the display microlens array 112 equipped on the display device 110. The ratio between the size of the image sensor 13 of the digital camera 1, which is the input member, and the size of the display 111 is defined as k. Below, we will explain the case where the shooting magnification (the ratio between the size of the subject and the size of the subject image) exceeds k and the case where it is k or less.

[0072] Let n be the magnification of the image. When the magnification exceeds k, that is, when the subject is located at a distance greater than a certain distance from the digital camera 1, the subject is captured by the image sensor 13 at 1 / n times the size. Therefore, the image is acquired two-dimensionally and displayed on the display 111 at a size k / n times larger. Since n>k, the subject appears smaller than it actually is, that is, it is closer to the display surface of the display 111. In the display device 110 of the present invention, a stereoscopic image is displayed in the vicinity of the surface of the display microlens array 112. If the distance to this display surface is d and the distance to the subject at the time of shooting is y, the magnification in the depth direction is expressed by the following equation (8).

number

[0073] The displayed image is k / n times larger, in other words (n / k) 2 It can also be said that the magnification is 50 times. If the shooting magnification is 50 times and the size of display 111 is 10 times that of image sensor 13, the magnification in the depth direction is 25 times, and an object 20 cm in the depth direction will have an image plane height of 80 μm on digital camera 1 and 8 mm on display 111. If the F-number of display microlens array 112 and the F-number of microlens array 12 when shooting with digital camera 1 are equal, the size of display 111 will be 10 times larger. As a result, an image plane height of only about 800 μm can be obtained on display 111.

[0074] Therefore, the F-number of the display microlenses 116 is increased to achieve consistency. However, it is unrealistic to set the F-number of the display microlenses 116 to 40 when the F-number of the microlenses 120 of the digital camera 1 is 4. Therefore, the display microlenses 116 are set to have an F-number of, for example, around 8 or 16. As a result, although the depth distance of the aerial image is somewhat reduced, the three-dimensional effect itself is not impaired. This is because the three-dimensional effect in human perception is relative and qualitative.

[0075] The following explains why an increase in the F-number of the display microlenses 116 leads to an increase in the depth of the aerial image. As shown in the relationship between the focal position and the surface of the display pixel 115 in Fig. 16, if the height of the optical image is y, the distance coordinate from the center (pseudo optical axis) of the display microlenses 116 is x, the spacing between the display microlenses 116 is d, and the focal length thereof is f, then the relationship shown in the following equation (9) holds. y / nd=f / x (9) Note that n is an integer that represents adjacent, and n=1 represents the adjacent display microlens 116, and n=2 represents the further adjacent display microlens 116.

[0076] The above equation (9) is equally valid for both the digital camera 1 and the display device 110. According to the relationship shown in equation (9), the height y of the optical image of the aerial image reproduced by the display 111 is proportional to the focal length f of the display microlenses 116. If the size of the display 111 is 10 times that of the image sensor 13, the focal length f will also be 10 times. Therefore, if the F-number is the same, an aerial image that is 10 times larger will be displayed at a height y that is 10 times larger on a 10 times larger display 111. Therefore, to further emphasize the height y, the focal length of the display microlenses 116 can be increased by 20 times, 30 times, etc., i.e., the F-number of the display microlenses 116 can be increased by two or three times.

[0077] Here, we will explain the case where the magnification is n / k>1. The sense of three-dimensionality is somewhat different from that of something far away. When the magnification is small at close range, that is, when taking close-up shots, the viewer does not expect the image to be life-size. For example, when considering a close-up shot of a bee, the viewer does not expect the bee to be displayed at its actual size on the display 111 or in three-dimensional form in the air. When a bee is displayed as an aerial image, it is natural for it to be about the size of a dove and located at some distance. This is because, when displayed at life-size, the viewer cannot tell whether the bee, which appears small, is a fly or a wasp. Furthermore, this is due to the fact that such objects have always been presented as enlarged images.

[0078] For example, if a subject is photographed at life-size using digital camera 1 with a 50 mm taking lens L1, the depth on the image plane is life-size and is saved as is. If this life-size photograph of the subject is output to the above-mentioned display 111 with a size of 10 times, a 20 cm bee will be displayed at a position of 500 mm. If the observation position of display 111 is 1 m, the observer will perceive the depth as 4 times larger, and the three-dimensional effect will be exaggerated.

[0079] As described above, the stereoscopic effect varies considerably depending on the magnification of the captured image or the displayed image. However, as mentioned above, the human sense of depth is vague, and the order of the depth of an object, rather than the absolute value, is the criterion for determining the stereoscopic effect. In the present invention, although the absolute value of the depth amount changes depending on the capture magnification, the relative relationship is completely maintained, so that a clear stereoscopic effect can be provided to the viewer. Furthermore, since this method provides a completely stereoscopic image and is fundamentally different from stereo methods that obtain a stereoscopic effect through parallax, it places less strain on the human visual sense and does not induce phenomena such as stereoscopic motion sickness.

[0080] Furthermore, the present invention is not limited to the above-described embodiments, and other embodiments that are conceivable within the scope of the technical concept of the present invention are also included within the scope of the present invention, as long as the characteristics of the present invention are not impaired. The embodiments and modifications used in the description may be configured by appropriately combining each other. [Explanation of symbols]

[0081] 100...imaging unit, 101...control circuit, 105...image integrating section, 106...image pattern generating section, 110...display device, 111...display device, 112...display microlens array, 113...virtual image lens, 114...display pixel array, 115...display pixel, 116...display microlens

Claims

[Claim 1] an input unit disposed corresponding to the imaging microlenses, to which a plurality of image signals output from a plurality of imaging pixels that receive light that has passed through the imaging microlenses are input; a plurality of arrayed display microlenses; a plurality of display pixels arranged for each of the display microlenses, each of which emits light to the display microlenses; a generation unit that generates display image data based on the plurality of image signals input to the input unit; a control unit that causes the display pixels to emit light based on the display image data, When the arrangement pattern of incident light of the imaging pixel corresponding to the photographing microlens is not the same as the arrangement pattern of light emitted from the display pixel corresponding to the display microlens, the generation unit assigns the image signal to the display pixel arranged at a position that is point-symmetrical with respect to the pseudo-optical axis of the microlens.

Images