Stereoscopic image generation device and program

Abstract

To present a stereoscopic image with less discomfort to a user.SOLUTION: A stereoscopic image generation device includes: a subject information acquisition part configured to acquire, via a network, subject information including image information obtained by capturing an image of a subject by a visible light sensor and distance information indicating a distance to the subject detected for each pixel captured by a distance sensor, the distance information having an information amount reduced by an encoding condition based on the image information, from an imaging part including the visible light sensor and the distance sensor disposed at a position corresponding to a pixel of the visible light sensor; a parallax setting information acquisition part configured to acquire parallax setting information according to the distance between left and right eyes of a user; a decoding part configured to decode the acquired distance information based on the image information; an image generation part configured to generate second image information indicating the image presented to the other eye of the user based on the parallax setting information, the decoded distance information, and the image information; and an output part configured to output first image information and the second image information in pairs as stereoscopic image information.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a stereoscopic image generating device and a program. [Background technology]

[0002] BACKGROUND ART Conventionally, an image display device has been disclosed that enables stereoscopic viewing by presenting a left eye image and a right eye image having a parallax between them to the left eye and right eye of a user (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2013-167791 Summary of the Invention [Problem to be solved by the invention]

[0004] In such an image display device, some users may experience discomfort due to the parallax between the left eye image and the right eye image (for example, discomfort similar to so-called "3D sickness"). However, the above-described conventional techniques have the problem that it is difficult to present a stereoscopic image that does not cause discomfort to the user. [Means for solving the problem]

[0005] One embodiment of the present invention is a stereoscopic image generating device comprising: a subject information acquisition unit that acquires, via a network, subject information from an imaging unit having a visible light sensor and a distance sensor arranged at a position corresponding to a pixel of the visible light sensor, the subject information including image information of a subject captured by the visible light sensor and information indicating the distance to the subject detected for each pixel captured by the distance sensor, the distance information having an amount of information reduced by predetermined encoding conditions based on the image information; a parallax setting information acquisition unit that acquires parallax setting information according to the distance between a user's left and right eyes; a decoding unit that decodes the acquired distance information based on the image information; an image generation unit that, when the image information is defined as first image information indicating an image to be presented to one of the user's left and right eyes, generates second image information indicating an image to be presented to the other eye of the user based on the parallax setting information, the decoded distance information, and the image information; and an output unit that pairs the first image information and the second image information and outputs them as stereoscopic image information.

[0006] One embodiment of the present invention is a program for causing a computer to execute the following steps: acquire, via a network, subject information from an imaging unit having a visible light sensor and a distance sensor positioned at a position corresponding to a pixel of the visible light sensor, the subject information including image information of a subject captured by the visible light sensor and information indicating the distance to the subject detected for each pixel captured by the distance sensor, the distance information having an amount of information reduced by predetermined encoding conditions based on the image information; acquire disparity setting information for the user's left and right eyes; decode the acquired distance information based on the image information; when the image information is defined as first image information indicating an image to be presented to one of the user's left and right eyes, generate second image information indicating an image to be presented to the other eye of the user based on the disparity setting information, the decoded distance information, and the image information; and output the first image information and the second image information as a pair as stereoscopic image information. [Effects of the Invention]

[0007] According to the present invention, it is possible to provide a stereoscopic image generating device and a program capable of presenting a stereoscopic image that gives a user a feeling of discomfort. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram illustrating an example of the configuration of a stereoscopic image generation system according to an embodiment of the present invention. [Figure 2] FIG. 2 is a diagram illustrating an example of the configuration of an imaging unit according to the present embodiment. [Figure 3] 1A and 1B are diagrams illustrating an example of a subject for which the amount of distance information is reduced in this embodiment. [Figure 4] FIG. 1 is a diagram illustrating an example of a functional configuration of a stereoscopic image generating device according to an embodiment of the present invention. [Figure 5] FIG. 2 is a diagram illustrating an example of a geometric positional relationship when a subject is viewed stereoscopically. [Figure 6] FIG. 2 is a diagram showing an example of the flow of operations of the stereoscopic image generating device of the present embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. 1 is a diagram showing an example of the configuration of a stereoscopic image generation system 1 according to this embodiment. The stereoscopic image generation system 1 includes an imaging unit 10, a transmission device 20, a stereoscopic image generation device 30, and a display device 40. Of these devices, the transmission device 20 and the stereoscopic image generation device 30 are connected via a network NT such as the Internet. Therefore, the device group consisting of the imaging unit 10 and the transmission device 20 and the device group consisting of the stereoscopic image generation device 30 and the display device 40 can be located in locations spatially separated from each other (i.e., remote locations).

[0010] The imaging unit 10 captures an image of a subject 500. In one example of this embodiment, the subject 500 includes various subjects, a first subject 501 to a sixth subject 506. In the following description, when these individual subjects (the first subject 501 to the sixth subject 506) are not distinguished from one another, they will be collectively referred to as the subject 500.

[0011] The subject 500 exists in a three-dimensional space indicated by the x-axis, y-axis, and z-axis. That is, the imaging unit 10 captures an image of the subject 500 existing in the three-dimensional space. In the following description, the direction of the x-axis is also referred to as the horizontal direction, the direction of the y-axis as the vertical direction, and the direction of the z-axis as the depth direction. An example of the configuration of the imaging unit 10 will be described with reference to FIG. 2.

[0012] 2 is a diagram showing an example of the configuration of the image capturing unit 10 of this embodiment. The image capturing unit 10 includes a lens 101, a dichroic mirror 102, a visible light sensor 111, an image information generating unit 112, a TOF sensor 121, a distance information generating unit 122, and a laser light source 130.

[0013] The lens 101 guides light incident from a predetermined angle range centered on the imaging axis AX to the dichroic mirror 102. In one example of this embodiment, the direction of the imaging axis AX coincides with the z-axis direction (i.e., the depth direction) of the space in which the subject 500 exists. When the subject 500 is present within a predetermined angle range, the lens 101 guides the light emitted from the subject 500 (for example, the light reflected by the subject 500) to the dichroic mirror .

[0014] The dichroic mirror 102 reflects light within a predetermined frequency range among the light guided from the lens 101, and transmits light within other predetermined frequency ranges. That is, the dichroic mirror 102 selectively reflects or transmits light guided from the lens 101. In one example of this embodiment, the dichroic mirror 102 reflects light in the visible light frequency range and guides it to the visible light sensor 111. The dichroic mirror 102 also transmits light in the infrared light frequency range.

[0015] The visible light sensor 111 includes, for example, a CMOS (Complementary Metal Oxide Semiconductor) image sensor, and detects light in the visible light frequency range reflected by the dichroic mirror 102 by separating it into three primary colors of R (red), G (green), and B (blue) for each pixel. The visible light sensor 111 outputs an intensity signal indicating the intensity of the detected light to the image information generation unit 112. The image information generation unit 112 includes a so-called image generation engine, and generates image information 510 based on the light intensity signal detected by the visible light sensor 111. This image information 510 is color image information expressed by a predetermined gradation value (e.g., 8 bits or 16 bits) for each of (R, G, B). The image information generation unit 112 outputs the generated image information 510 to the transmission device 20.

[0016] The laser light source 130 emits laser light in a predetermined frequency range in approximately the same direction as the direction of the imaging axis AX of the lens 101 (i.e., the depth direction). In one example of this embodiment, the laser light source 130 emits light in the infrared frequency range. As described above, the dichroic mirror 102 in one example of this embodiment transmits light in the infrared frequency range. That is, the light emitted from the laser light source 130 is reflected by the subject 500 and enters the lens 101, then transmits through the dichroic mirror 102 and is guided to the TOF sensor 121.

[0017] The TOF sensor 121 (a time-of-flight sensor; an example of a distance sensor) detects light in a frequency domain (for example, light in an infrared frequency domain) that is incident from the lens 101 and transmitted through the dichroic mirror 102. As described above, when laser light in the infrared frequency domain emitted from the laser light source 130 is reflected by the subject 500 and incident on the lens 101, it passes through the dichroic mirror 102 and is guided to the TOF sensor 121. That is, the TOF sensor 121 detects the laser light reflected by the subject 500. The TOF sensor 121 outputs a signal indicating the detection result of the laser light to the distance information generation unit 122.

[0018] Distance information generation unit 122 calculates the distance between subject 500 and imaging unit 10 based on a signal indicating the detection result output by TOF sensor 121. Distance information generation unit 122 outputs distance information 520 indicating the calculated distance to transmission device 20.

[0019] Here, when the laser light source 130 emits pulsed light toward the subject 500, the time from when the pulsed light is emitted from the laser light source 130 until it is reflected by the subject 500 and detected by the TOF sensor 121 can be measured. The distance information generating unit 122 measures the time from when the laser light source 130 emits the pulsed light until when the TOF sensor 121 detects it. The distance information generating unit 122 calculates the distance between the subject 500 and the TOF sensor 121 based on the measured time and the traveling speed of the pulsed light (i.e., the speed of light). In this way, the TOF sensor 121 functions as a distance sensor that detects the distance from the imaging unit 10 to the subject 500.

[0020] The visible light sensor 111 and the TOF sensor 121 of this embodiment are disposed symmetrically with respect to the imaging axis AX, with the dichroic mirror 102 sandwiched between them. That is, the visible light sensor 111 and the TOF sensor 121 are disposed substantially coaxially. Therefore, if the pixels of the TOF sensor 121 of this embodiment are disposed in the same manner as the pixels of the visible light sensor 111, the TOF sensor 121 can measure the distance to the subject 500 for each pixel of the visible light sensor 111.

[0021] That is, the imaging unit 10 includes a visible light sensor 111 and a TOF sensor 121 (distance sensor) arranged at a position corresponding to a pixel of the visible light sensor 111. The number of pixels of the TOF sensor 121 does not have to match the number of pixels of the visible light sensor 111. In this case, for pixels in the TOF sensor 121 that do not correspond to those in the visible light sensor 111, an interpolation process may be performed using adjacent pixels. This interpolation process may be performed by the imaging unit 10 or the transmission device 20, or may be performed by the stereoscopic image generation device 30.

[0022] 1, the transmission device 20 acquires the image information 510 and the distance information 520 output by the imaging unit 10. The transmission device 20 transmits the image information 510 and the distance information 520 to the stereoscopic image generation device 30 via the network NT.

[0023] Here, transmitting device 20 may reduce the amount of information in distance information 520 by encoding based on a predetermined information amount reduction algorithm. In the following description, distance information 520 encoded by transmitting device 20 is also referred to as encoded distance information 521. An example of the information amount reduction algorithm used by transmitting device 20 will be described.

[0024] 3 is a diagram showing an example of a subject 500 for which the amount of information in the distance information 520 of this embodiment is to be reduced. The figure shows a case in which a subject image P500 includes a first subject image P501 to a sixth subject image P506.

[0025] In this example, the first object image P501 is an image in which the movement within the object image P500 is greater than that within the other object images 500. The second object image P502 and the third object image P503 are images with relatively little movement. The fourth object image P504 is an image located closer to the front in the depth direction than the fifth object image P505 and the sixth object image P506.

[0026] (1) When selecting a subject with large movement as the target for stereoscopic viewing Generally, a subject 500 that moves a lot may stand out more in the subject image P500. For this reason, a subject 500 that moves a lot may be more preferable to be used as a target for stereoscopic viewing than a subject 500 that moves less.

[0027] Transmitting device 20 may select subject 500 with large movement as a target for stereoscopic viewing. In this case, transmitting device 20 may transmit distance information 520 of subject 500 with large movement, and may exclude distance information 520 of subject 500 with small movement from the target for transmission.

[0028] In the example shown in the figure, the transmitting device 20 transmits distance information 520 about a first object image P501, which is an image with greater movement within the object image P500 than the other objects 500. The transmitting device 20 also excludes distance information 520 about the second object image P502 to the sixth object image P506, which are images with less movement than the first object image P501, from the transmission target. Alternatively, the amount of distance information 520 may be reduced by reducing the transmission frequency by transmitting the distance information 520 about the second object image P502 to the sixth object image P506 every predetermined number of frames.

[0029] According to the stereoscopic image generation system 1 configured in this manner, the distance information 520 received by the stereoscopic image generation device 30 can be reduced compared to when distance information 520 for all of the subjects 500 in the subject image P500 is transmitted.

[0030] Here, whether or not distance information 520 of a certain subject 500 is to be transmitted can also be called an encoding condition for distance information 520.

[0031] In other words, the encoding conditions include reducing the amount of information in distance information 520 by transmitting distance information 520 for subjects 500 with relatively large movements and excluding distance information 520 for subjects 500 with relatively small movements.

[0032] (2) When selecting a subject within a specified distance range as the target for stereoscopic viewing In general, a subject 500 that is closer in the depth direction (i.e., a subject 500 that is closer to the viewer) may stand out more in the subject image P500. Also, a subject 500 that is within a certain range in the depth direction (for example, a distance that is neither too close nor too far from the viewer) may stand out more in the subject image P500. In such a case, it may be preferable to use subjects 500 that are within a predetermined distance range in the depth direction as the target of stereoscopic viewing.

[0033] Transmitting device 20 may select subjects 500 within a predetermined distance range as targets for stereoscopic viewing. In this case, transmitting device 20 sets distance information 520 of subjects 500 within the predetermined distance range as targets for transmission, and sets distance information 520 of subjects 500 outside the predetermined distance range as targets for transmission. Alternatively, the amount of distance information 520 may be reduced by reducing the frequency of transmission of distance information 520 of subjects 500 outside the predetermined distance range every predetermined number of frames.

[0034] In the example shown in the figure, the transmitting device 20 transmits the distance information 520 for the fourth object image P504, which is the foremost of the fourth object image P504, the fifth object image P505, and the sixth object image P506. The transmitting device 20 also excludes the distance information 520 for the fifth object image P505 and the sixth object image P506 from the transmission targets.

[0035] According to the stereoscopic image generation system 1 configured in this manner, the distance information 520 received by the stereoscopic image generation device 30 can be reduced compared to when distance information 520 for all of the subjects 500 in the subject image P500 is transmitted.

[0036] In other words, the encoding conditions include reducing the amount of information in the distance information 520 by transmitting distance information 520 for subjects 500 within a predetermined distance range from the imaging unit 10, and excluding distance information 520 outside the distance range from the distance information 520.

[0037] In the examples of (1) and (2) above, a method was described in which the amount of distance information 520 received by the stereoscopic image generating device 30 is reduced by selecting a target to which distance information 520 is to be sent from among multiple subjects 500. The stereoscopic image generation system 1 may reduce the amount of information in the distance information 520 received by the stereoscopic image generation device 30 by compressing the amount of information in the distance information 520 by devising a method for encoding the distance information 520. An example of a method for encoding the distance information 520 will be described below.

[0038] (3) Compression of information volume by run-length coding Generally, there may be an area in the subject image P500 where the distance between adjacent pixels does not change between the imaging unit 10 and the subject 500. For example, if the subject image P500 is an image of an interior of a room, the distance from the wall of the room to the imaging unit 10 does not change much, and there is an area where the distance between adjacent pixels does not change. In such a case, the transmitting device 20 reduces the amount of information in the distance information 520 by using a known run-length encoding technique. That is, the transmitting device 20 encodes, into the distance information 520, information indicating that the distance between adjacent pixels has not changed. For example, in an area where pixels with the same distance are consecutive, the transmitting device 20 generates encoded distance information 521 in which the distance of each pixel is replaced with the number of times the distance is repeated.

[0039] That is, the amount of information in the encoded distance information 521 (distance information) is reduced by run-length encoding.

[0040] (4) Compression of information volume by relative distance As described above, there may be an area in the subject image P500 where the distance between adjacent pixels does not change between the imaging unit 10 and the subject 500. For example, if the subject image P500 is an image of an interior of a room, the distance from the wall of the room to the imaging unit 10 does not change much, and there is an area where the distance between adjacent pixels does not change. In such a case, the transmitting device 20 calculates the absolute distance, which is the distance between the imaging unit 10 and the subject 500, for a representative pixel in an area where the distance between adjacent pixels does not change, and calculates the relative distance from the distance of the representative pixel for other pixels. In a region where the distance between adjacent pixels does not change, the relative distance between adjacent pixels may be sufficiently small compared to the absolute distance between each pixel. In such a case, the gradation value required to express the relative distance can be made smaller than the gradation value required to express the absolute distance (e.g., the number of bits required for digital expression). For example, if the gradation value of the absolute distance is expressed in 16 bits, the gradation value of the relative distance may be expressed in 8 bits.

[0041] In such a case, the amount of information can be reduced by expressing the distances of some pixels in the subject 500 as relative distances to the distances of other pixels, compared to expressing the distances of all pixels in the subject image P500 as absolute distances. The transmitting device 20 generates encoded distance information 521 using relative distance information that indicates the distance to the subject 500 based on the relative distance.

[0042] In other words, the encoded distance information 521 (distance information) is information with fewer gradation values ​​than absolute distance information that indicates the distance to the subject 500 based on absolute distance, and the amount of information is reduced by using relative distance information that indicates the distance to the subject 500 based on relative distance to the absolute distance.

[0043] Returning to FIG. 1, the transmission device 20 transmits the image information 510 and the encoded distance information 521 encoded by the encoding method described above to the stereoscopic image generation device 30 via the network NT. Stereoscopic image generation device 30 receives image information 510 and encoded distance information 521 transmitted by transmission device 20 via network NT. The functional configuration of this stereoscopic image generation device 30 will be described with reference to FIG.

[0044] [Functional configuration of the stereoscopic image generation device] FIG. 4 is a diagram showing an example of the functional configuration of the stereoscopic image generating device 30 of this embodiment. Stereoscopic image generating device 30 includes a calculation unit 310 and a storage unit 320. The calculation unit 310 includes, for example, a CPU (Central Processing Unit) and provides various functions based on programs and data stored in storage unit 320, which is a non-volatile storage unit.

[0045] The calculation unit 310 includes, as its functional units, a subject information acquisition unit 311, a parallax setting information acquisition unit 312, a decoding unit 313, an image generation unit 314, and an output unit 315.

[0046] The subject information acquisition unit 311 receives the image information 510 and the encoded distance information 521 transmitted by the transmission device 20 via the network NT, and acquires them as subject information. As described above, the imaging unit 10 is located in a remote location via the network NT from the user viewing the display device 40. Therefore, the subject 500 captured by the imaging unit 10 cannot be directly viewed by the user. In other words, the image information 510 is information about an image captured by the visible light sensor 111 of the subject 500 that cannot be directly viewed by the user. In addition, the encoded distance information 521 is information indicating the distance to the subject 500 detected for each pixel by the TOF sensor 121 (distance sensor), and is distance information 520 in which the amount of information has been reduced by predetermined encoding conditions based on the image information 510.

[0047] That is, the subject information acquisition unit 311 receives information via the network NT, including image information 510 of the subject 500 captured by the visible light sensor 111, and distance information 520 (encoded distance information 521), which is information indicating the distance to the subject 500 detected for each pixel by the TOF sensor 121 (distance sensor), and in which the amount of information has been reduced by predetermined encoding conditions based on the image information 510, and acquires it as subject information. Note that subject 500 may not be directly visible to the user. In this case, it can be said that subject information acquisition unit 311 receives information including image information 510 of subject 500 captured by visible light sensor 111 and the above-mentioned distance information 520 (encoded distance information 521) via network NT and acquires it as subject information.

[0048] The object information acquisition unit 311 outputs the acquired image information 510 to the decoding unit 313, the image generation unit 314, and the output unit 315. The object information acquisition unit 311 also outputs the acquired encoded distance information 521 to the decoding unit 313.

[0049] The parallax setting information acquisition unit 312 acquires the parallax setting information 301 according to the distance between the left and right eyes of the user. For example, the parallax setting information acquisition unit 312 is connected to the operation unit 31. The operation unit 31 includes a dial, a switch, a touch panel, etc. that can be operated by the user, and sets the parallax setting information 301 according to the distance between the left and right eyes specific to the user. Here, the geometric positional relationship when viewing a subject stereoscopically will be described with reference to FIG. 5.

[0050] 5 is a diagram showing an example of the geometric positional relationship when viewing a subject stereoscopically. In the figure, light emitted from subject 500 reaches image capture unit 10, which is a distance Z away in the depth direction, and forms an image at focal length F of lens 101. Image capture unit 10 captures an image of subject 500 by placing an imaging surface (for example, visible light sensor 111) at the position where the light is formed. Light emitted from subject 500 also reaches a position that is separated from imaging unit 10 in the horizontal direction (x-axis direction in FIG. 1) by inter-camera distance B. If a virtual visible light sensor is placed at this position, light from subject 500 enters this virtual visible light sensor at a position that is separated by parallax D from the position at which light enters visible light sensor 111. Here, if the visible light sensor 111 shown in the same figure is considered to be an image sensor for the left eye image PL and the virtual visible light sensor is considered to be an image sensor for the right eye image PR, a parallax D is expressed between the left eye image PL and the right eye image PR generated by these image sensors. When the user views the left-eye image PL with his left eye and the right-eye image PR with his right eye, the user can recognize a stereoscopic image of the subject 500 due to the parallax D between these images.

[0051] However, if the inter-camera distance B does not match the distance between the user's left and right eyes, the magnitude of the parallax D between the left eye image PL and the right eye image PR will differ from the magnitude of the parallax when the user views the subject 500 directly. In this case, the user may not be able to see the left eye image PL and the right eye image PR naturally, and may feel uncomfortable. Therefore, it is preferable that the inter-camera distance B matches the distance between the user's left and right eyes.

[0052] Here, the imaging unit 10 of this embodiment is equipped with only a visible light sensor 111 (corresponding to the left eye image sensor in the same figure), and does not have a right eye image sensor in the same figure. On the other hand, the imaging unit 10 of this embodiment is equipped with a TOF sensor 121, and is capable of outputting distance information 520.

[0053] As shown in the figure, the relationship of the following formula (1) holds among the distance Z between the subject 500 and the imaging unit 10, the inter-camera distance B, the focal length F of the imaging unit 10, and the parallax D.

[0054] Z = (B × F) / D … (1)

[0055] Of the parameters shown in equation (1), distance Z is known because distance information 520 (or encoded distance information 521) can be acquired. Furthermore, focal length F of image capture unit 10 is known because it is determined based on the optical design of image capture unit 10. Therefore, disparity D can be calculated by setting inter-camera distance B according to the distance between the user's left and right eyes.

[0056] 4 , the operation unit 31 accepts an operation for setting an inter-camera distance B corresponding to the distance between the user's left and right eyes. The parallax setting information acquisition unit 312 acquires the inter-camera distance B set by the operation unit 31 as parallax setting information 301. The parallax setting information acquisition unit 312 outputs the acquired parallax setting information 301 to the image generation unit 314.

[0057] The decoding unit 313 generates decoded distance information 522 based on the image information 510 and encoded distance information 521 output by the object information acquisition unit 311. The decoding unit 313 outputs the generated decoded distance information 522 to the image generation unit 314.

[0058] That is, the decoding unit 313 decodes the acquired distance information 520 (encoded distance information 521) based on the image information 510. If the number of pixels of the TOF sensor 121 does not match the number of pixels of the visible light sensor 111, the decoding unit 313 may perform the above-described interpolation process on the decoded distance information 522.

[0059] The image generation unit 314 calculates the parallax D based on the above-described distance Z, focal length F, and inter-camera distance B. The image generation unit 314 applies the calculated parallax D to the image information 510 to generate an image (for example, a right-eye image PR) having the parallax D with respect to the image (for example, a left-eye image PL) indicated by the image information 510.

[0060] That is, when image information 510 is first image information 511 indicating an image to be presented to one of the user's left and right eyes, image generation unit 314 generates second image information 512 indicating an image to be presented to the user's other eye based on disparity setting information 301, decoded distance information 522 (decoded distance information), and image information 510.

[0061] The stereoscopic image generating device 30 configured in this manner can present images (first image information 511, second image information 512) having a parallax D according to the distance between the left and right eyes specific to the user. Here, when the imaging unit includes a sensor for a left eye image and a sensor for a right eye image and outputs an image for the left eye and an image for the right eye, it is not possible to provide a parallax D according to the distance between the left and right eyes that is specific to the user. If an imaging unit with such a configuration were used to provide a parallax D according to the distance between the left and right eyes that is specific to the user, it would be necessary to physically (mechanically) change the distance between the sensors included in the imaging unit, which would be troublesome.

[0062] According to stereoscopic image generating device 30 of this embodiment, an image including disparity D is generated for image information 510 based on distance information 520 (or decoded distance information 522). Therefore, according to stereoscopic image generating device 30 of this embodiment, it is possible to provide disparity D according to the distance between the left and right eyes specific to the user while eliminating the above-mentioned complexity. That is, the stereoscopic image generating device 30 of this embodiment can reduce the discomfort felt by the user when viewing the subject 500 stereoscopically.

[0063] The image generating unit 314 outputs the generated second image information 512 to the output unit 315.

[0064] The output unit 315 outputs the image information 510 output by the subject information acquisition unit 311 as first image information 511 (e.g., image information of the left eye image PL) and the second image information 512 generated by the image generation unit 314 as, e.g., image information of the right eye image PR to the display device 40.

[0065] That is, the output unit 315 outputs the first image information 511 and the second image information 512 as a pair, as stereoscopic image information 513.

[0066] The display device 40 displays the left-eye image PL and the right-eye image PR based on the stereoscopic image information 513 output by the output unit 315. The user can recognize a stereoscopic image of the subject 500 by looking at the left eye image PL and the right eye image PR displayed on the display device 40.

[0067] [Operational flow of stereoscopic image generation device 30] FIG. 6 is a diagram showing an example of the flow of operations of the stereoscopic image generating device 30 of this embodiment. (Step S10) The subject information acquisition unit 311 acquires the image information 510 and the distance information 520 (specifically, the encoded distance information 521) from the transmission device 20. (Step S20) The parallax setting information acquisition unit 312 acquires the parallax setting information 301 from the operation unit 31. (Step S30) The decoding unit 313 decodes the encoded distance information 521 acquired in step S10 using a predetermined decoding procedure. The image generation unit 314 generates second image information 512 based on the decoded distance information 522, which is the decoded distance information, and the image information 510 acquired in step S10. (Step S40) The output unit 315 outputs stereoscopic image information 513, in which the image information 510 is the first image information 511 (for example, image information of the left eye image PL) and the second image information 512 generated in step S30 is the image information of the right eye image PR, to the display device 40. The display device 40 displays the left eye image PL and the right eye image PR based on the stereoscopic image information 513.

[0068] The subject information acquisition unit 311 receives information from the imaging unit 10, which is equipped with a visible light sensor 111 and a TOF sensor 121 (distance sensor) that is arranged at a position corresponding to the pixel of the visible light sensor 111 and measures distance based on the reflection time of the emitted distance measurement light, via the network NT, and acquires the information as subject information. The information includes image information 510 of a subject 500 that is not directly visible to the user, captured by the visible light sensor 111, and distance information 520 that indicates the distance to the subject 500 detected for each pixel by the TOF sensor 121 (distance sensor).

[0069] Here, the TOF sensor 121 detects light emitted from the laser light source 130 and reflected by the subject 500. For this reason, the TOF sensor 121 cannot obtain distance information for an area where light is not reflected. Conversely, the TOF sensor 121 can distinguish between an area where the subject 500 exists and an area where the subject 500 does not exist within the space to be measured for distance.

[0070] Distance information 520 output by TOF sensor 121 is distance information about the area where subject 500 exists. Therefore, with TOF sensor 121, the amount of information in distance information 520 can be reduced compared to when distance information is acquired about all areas of the space that are the target of distance measurement. That is, according to the stereoscopic image generating system 1 of this embodiment, by using the TOF sensor 121, the amount of information in the distance information 520 can be reduced.

[0071] Although the embodiments of the present invention have been described in detail above with reference to the drawings, the specific configuration is not limited to this embodiment and can be appropriately modified without departing from the spirit of the present invention. The configurations described in the above-described embodiments may be combined.

[0072] Each unit included in each device in the above-described embodiments may be realized by dedicated hardware, or may be realized by a memory and a microprocessor.

[0073] In addition, each part of each device may be composed of a memory and a CPU (central processing unit), and the functions of each part of each device may be realized by loading a program into memory and executing it.

[0074] In addition, a program for realizing the functions of each unit of each device may be recorded on a computer-readable recording medium, and the program recorded on the recording medium may be read into a computer system and executed to perform processing by each unit of the control unit. Note that the term "computer system" here includes hardware such as an OS and peripheral devices.

[0075] Furthermore, if a WWW system is used, the "computer system" also includes the homepage provision environment (or display environment). "Computer-readable recording media" refers to portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, as well as storage devices such as hard disks built into computer systems. Furthermore, "computer-readable recording media" also includes devices that dynamically store programs for a short period of time, such as communication lines used when transmitting programs over networks like the Internet or over communication lines like telephone lines, and devices that store programs for a fixed period of time, such as volatile memory within computer systems that serve as servers or clients. Furthermore, the programs may be programs that implement some of the aforementioned functions, or may be programs that can realize the aforementioned functions in combination with programs already stored in the computer system. [Explanation of symbols]

[0076] 1...stereoscopic image generating system, 10...imaging unit, 101...lens, 102...dichroic mirror, 111...visible light sensor, 112...image information generating unit, 121...TOF sensor, 122...distance information generating unit, 20...transmitting device, 30...stereoscopic image generating device, 31...operation unit, 301...parallax setting information, 310...calculating unit, 320...storage unit, 311...subject information acquiring unit, 312...parallax setting information acquiring unit unit, 313...decoding unit, 314...image generating unit, 315...output unit, 40...display device, 500...object, 510...image information, 511...first image information, 512...second image information, 520...distance information, 521...encoded distance information, 522...decoded distance information, P500...object image, PL...left eye image (first image), PR...right eye image (second image), AX...imaging axis, NT...network

Claims

1. a subject information acquisition unit that acquires, via a network, subject information from an imaging unit that includes a visible light sensor and a distance sensor disposed at a position corresponding to a pixel of the visible light sensor, the subject information including image information of a subject captured by the visible light sensor and information indicating a distance to the subject detected for each pixel captured by the distance sensor, the distance information being reduced in amount by a predetermined encoding condition based on the image information; a parallax setting information acquisition unit that acquires parallax setting information according to the distance between the left and right eyes of the user; a decoding unit that decodes the acquired distance information based on the image information; an image generating unit that generates, when the image information is first image information indicating an image to be presented to one of the left and right eyes of the user, second image information indicating an image to be presented to the other eye of the user based on the parallax setting information, the decoded distance information, and the image information; an output unit that outputs the first image information and the second image information as a pair of stereoscopic image information; A stereoscopic image generating device comprising:

2. The encoding condition includes reducing the amount of distance information by setting the distance information for subjects within a predetermined distance range from the imaging unit to be transmitted and setting the distance information outside the distance range to be not transmitted. The stereoscopic image generating device according to claim 1 .

3. The encoding conditions include reducing the amount of distance information by transmitting the distance information about subjects with relatively large movements and excluding the distance information about subjects with relatively small movements. The stereoscopic image generating device according to claim 1 .

4. The distance information has fewer gradation values ​​than absolute distance information that indicates the distance to the subject by an absolute distance due to the amount of information being reduced by run-length encoding, and the amount of information is reduced by using relative distance information that indicates the distance to the subject by a distance relative to the absolute distance. The stereoscopic image generating device according to claim 1 .

5. On the computer, acquiring, via a network, from an imaging unit equipped with a visible light sensor and a distance sensor arranged at a position corresponding to a pixel of the visible light sensor, subject information including image information of a subject captured by the visible light sensor and information indicating a distance to the subject detected for each pixel by the distance sensor, the distance information being reduced in amount by a predetermined encoding condition based on the image information; Acquiring disparity setting information for the left and right eyes of a user; decoding the obtained distance information based on the image information; generating second image information indicating an image to be presented to the other eye of the user based on the parallax setting information, the decoded distance information, and the image information, when the image information is first image information indicating an image to be presented to one of the left and right eyes of the user; outputting the first image information and the second image information as a pair as stereoscopic image information; A program to execute.

Images