Information processing device, information processing method, and computer-readable recording medium

Abstract

An information processing device according to one embodiment of the present technology is equipped with a camera control unit. The camera control unit, on the basis of viewing parameters that include at least one of a pupillary distance of a viewer viewing a stereoscopic display device that displays in real time a stereoscopic image captured using a stereo camera, a viewing height of the viewer, and a viewing distance of the viewer, mechanically controls the stereo camera so as to adjust an imaging point of view of the stereo camera.

Description

Information processing device, information processing method, and computer-readable recording medium

[0001] The present technology relates to an information processing device, an information processing method, and a computer-readable recording medium that can be applied to a system that displays live stereoscopic images in real time.

[0002] Conventionally, systems have been developed that display images of remote locations in real time to enable communication, etc. For example, Patent Document 1 describes a remote communication system that transmits and receives images and audio in real time between two communication terminals in remote locations. In this system, a person image synthesis process is performed to synthesize an image of the user's face facing forward with high accuracy based on an image of the user taken from diagonally above. By transmitting this synthesized image to the other party, communication can be carried out while maintaining eye contact (see, for example, paragraphs

[0017] and

[0024] of the specification of Patent Document 1, and Figure 1).

[0003] International Publication No. 2019 / 026598

[0004] In recent years, technology has been developed to deliver live-action stereoscopic images in real time, and it is expected that this technology will be applied to communication tools, etc. For example, by providing a stereoscopic display that matches the viewer's conditions, such as their interpupillary distance and viewing position, viewers can experience communication with a sense of reality, as if the other party were right in front of them.

[0005] On the other hand, stereoscopic display that does not match the viewer's condition may cause the viewer to feel uncomfortable. In such cases, methods such as image processing to correct the stereoscopic image may be considered, but there are limits to what can be achieved by image processing alone, and the viewer may still feel uncomfortable or the image quality may deteriorate. For this reason, there is a demand for technology that realizes appropriate stereoscopic display that matches the viewer's condition.

[0006] In view of the above circumstances, an object of the present technology is to provide an information processing device, an information processing method, and a computer-readable recording medium that realize appropriate stereoscopic display that suits the state of the viewer.

[0007] To achieve the above object, an information processing device according to one embodiment of the present technology includes a camera control unit that mechanically controls the stereo cameras to adjust shooting viewpoints of the stereo cameras based on viewing parameters including at least one of an interpupillary distance of a viewer viewing a stereoscopic display device that displays stereoscopic images captured by the stereo cameras in real time, a viewing height of the viewer, and a viewing distance of the viewer.

[0008] In this information processing device, the stereo camera that captures the stereoscopic images is mechanically controlled based on the viewing parameters of the viewer viewing the stereoscopic images, such as the interpupillary distance, viewing height, viewing distance, etc. This makes it possible to adjust the shooting viewpoint of the stereo camera according to the viewer's condition, and to realize an appropriate stereoscopic display that matches the viewer's condition.

[0009] The stereo camera may be connected to an adjustment mechanism that mechanically changes at least one of an inter-camera distance of the stereo camera, a height position of the stereo camera, and a depth position of the stereo camera.

[0010] The viewing parameters may include an interpupillary distance of the viewer. In this case, the adjustment mechanism may be capable of adjusting an inter-camera distance of the stereo camera. Furthermore, the camera control unit may control the adjustment mechanism to adjust the inter-camera distance based on the interpupillary distance of the viewer.

[0011] The viewing parameters may include a viewing height of the viewer. In this case, the adjustment mechanism may be capable of adjusting a height position of the stereo camera. Furthermore, the camera control unit may control the adjustment mechanism to adjust the height position based on the viewing height of the viewer.

[0012] The viewing parameters may include a viewing distance of the viewer. In this case, the adjustment mechanism may be capable of adjusting a depth position of the stereo camera. Furthermore, the camera control unit may control the adjustment mechanism to adjust the depth position based on the viewing distance of the viewer.

[0013] The information processing device may further include an image generation unit that generates, in real time, the stereoscopic image to be displayed on the stereoscopic display device from the images captured by the stereo camera with the adjusted shooting viewpoint.

[0014] The image generation unit may generate the stereoscopic image by performing a shift process to adjust parallax, a trimming process to adjust the angle of view to a display range of the stereoscopic display device, and an enlargement process to adjust the image to a predetermined image size on the images captured by the stereo camera.

[0015] The scale of the stereoscopic image displayed by the stereoscopic display device may be set to life-size. In this case, the camera control unit may control the adjustment mechanism so that a positional relationship between a virtual display surface, which is an image of a display surface of the stereoscopic display device in an imaging space of the stereo camera, and an imaging viewpoint of the stereo camera becomes the same as a positional relationship between a display surface of the stereoscopic display device and a viewpoint of the viewer.

[0016] The viewing parameters may include the interpupillary distance of the viewer, the viewing height of the viewer, and the viewing distance of the viewer. In this case, the stereo camera may be adjustable in terms of its inter-camera distance, height position, and depth position. Furthermore, the camera control unit may match the inter-camera distance of the stereo camera to the interpupillary distance of the viewer, match the height position of the stereo camera to the viewing height of the viewer, and match the distance between the stereo camera and the virtual display surface to the viewing distance of the viewer.

[0017] The image generating section may shift the images captured by the stereo camera so that a parallax in the depth direction of the virtual display surface becomes zero.

[0018] The image generating section may trim an area corresponding to the virtual display surface as seen from the adjusted shooting viewpoint from the shifted image, and enlarge the trimmed image to a predetermined image size.

[0019] The adjustment mechanism may include a position sensor that detects a position of a shooting viewpoint of the stereo camera. In this case, the image generator may adjust the images captured by the stereo camera based on a difference between an adjustment value of the shooting viewpoint corresponding to the viewing parameters and a detection value of the position sensor.

[0020] The viewing parameters may include a viewing distance of the viewer. In this case, the image generator may shift the images captured by the stereo camera so that a parallax in the depth direction of the virtual display surface corresponds to the viewing distance.

[0021] The scale of the stereoscopic image displayed by the stereoscopic display device may be X times the actual size, where X is a positive real number. In this case, the viewing parameters may include the interpupillary distance of the viewer. The camera control unit may also set the inter-camera distance of the stereo cameras to 1 / X times the interpupillary distance of the viewer.

[0022] The viewing parameters may include a viewing height of the viewer. In this case, the camera control unit may set a height position of the stereo camera to a position different from the viewing height of the viewer.

[0023] The viewing parameters may be detected by sensors measuring the audience.

[0024] The viewing parameters may be input via a predetermined UI.

[0025] An information processing method according to one embodiment of the present technology is an information processing method executed by a computer system, and includes mechanically controlling a stereo camera to adjust a shooting viewpoint of the stereo camera based on viewing parameters including at least one of an interpupillary distance of a viewer viewing a stereoscopic display device that displays stereoscopic images captured using the stereo camera in real time, a viewing height of the viewer, and a viewing distance of the viewer.

[0026] A computer-readable recording medium according to one embodiment of the present technology stores a program for causing a computer system to execute the following steps: mechanically controlling the stereo cameras to adjust the shooting viewpoints of the stereo cameras based on viewing parameters including at least one of an interpupillary distance of a viewer viewing a stereoscopic display device that displays stereoscopic images captured using the stereo cameras in real time, a viewing height of the viewer, and a viewing distance of the viewer.

[0027] 1 is a schematic diagram showing a configuration example of a telepresence system according to a first embodiment of the present technology. FIG. 2 is a block diagram showing a configuration example of a 3D telepresence system. FIG. 3 is a schematic perspective view showing a configuration example of an imaging device. FIG. 4 is a schematic diagram showing an example of stereo camera images. FIG. 5 is a schematic diagram showing adjustment parameters for the imaging viewpoint of the stereo camera. FIG. 6 is a schematic diagram showing viewing parameters of a viewer. FIG. 7 is a schematic diagram showing an example of stereoscopic images. FIG. 8 is a schematic diagram for explaining stereoscopic display by a stereoscopic display device. FIG. 9 is a flowchart showing an example of operation of a 3D telepresence system. FIG. 10 is a schematic diagram showing an example of adjustment of the inter-camera distance of the stereo camera. FIG. 11 is a schematic diagram showing an example of adjustment of the height position of the stereo camera. FIG. 12 is a schematic diagram showing an example of adjustment of the depth position of the stereo camera. FIG. 13 is a schematic diagram showing an example of image processing processing. FIG. 14 is a schematic diagram showing a configuration example of a 3D telepresence system according to a third embodiment. FIG. 15 is a schematic diagram showing an example of parallax adjustment processing by image shifting. FIG. 16 is a schematic diagram showing a configuration example of a 3D telepresence system according to a fourth embodiment. 10 is a schematic diagram showing an example of adjusting the height position of a stereo camera. FIG. 11 is a schematic diagram showing an example of the configuration of a communication device using a half mirror.

[0028] Hereinafter, embodiments of the present technology will be described with reference to the drawings.

[0029] [Outline of 3D Telepresence System] Fig. 1 is a schematic diagram showing an example configuration of a telepresence system according to a first embodiment of the present technology. The 3D telepresence system 100 is a system that displays, in real time, stereoscopic video 5 captured using a stereo camera 10 on a stereoscopic display device 40. In other words, the 3D telepresence system 100 is a 3D display system that captures, transmits, and displays live-action stereoscopic video 5 in real time.

[0030] The upper part of Fig. 1 shows a schematic diagram of a shooting-side base where a stereo camera 10 is installed, and the lower part of Fig. 1 shows a schematic diagram of a viewing-side base where a stereoscopic display device 40 is installed. Hereinafter, three mutually orthogonal directions at the shooting-side and viewing-side bases will be referred to as the X direction, Y direction, and Z direction. The X direction corresponds to the height direction (vertical direction). The Y direction and Z direction are directions that are mutually orthogonal in a horizontal plane. The upper and lower diagrams of Fig. 1 show the views of each base as viewed along the Y direction.

[0031] The stereo camera 10 is a 3D imaging device that captures images that become the stereoscopic image 5. The stereoscopic display device 40 is a display device that displays the stereoscopic image 5 so that the viewer perceives a stereoscopic image 6. The stereoscopic image 5 includes a right-eye image that is displayed on the viewer's right eye and a left-eye image that is displayed on the viewer's left eye.

[0032] The following describes a configuration in which a stereoscopic image 6 captured of a user 1 at a shooting location is displayed to a user 2 at a viewing location. Specifically, the user 1 is captured by a stereo camera 10 at the shooting location. At the viewing location, a stereoscopic video image 5 captured of the user 1 is displayed by a stereoscopic display device 40. This allows the user 2 viewing the stereoscopic display device 40 to perceive the stereoscopic image 6 of the user 1.

[0033] The system may be configured so that user 1 can check the state of user 2. For example, a camera that captures the state of user 2 may be installed at the viewing location, and the image from the camera may be displayed on a display device installed at the capturing location. In this case, the image captured of user 2 does not need to be a stereoscopic image. Hereinafter, user 1 who is the subject of stereo camera 10 will be referred to as performer 1, and user 2 who watches stereoscopic display device 40 will be referred to as viewer 2.

[0034] The 3D telepresence system 100 according to this embodiment provides a life-size stereoscopic display with a sense of seamless connection. This type of stereoscopic display makes the display (the display surface 41 of the stereoscopic display device 40) look like a window, creating the illusion that another world or space is connected behind the display. This effect is extremely important in providing a realistic experience to the viewer 2.

[0035] In order to capture images with this sense of depth, at the filming location, a stereo camera 10 is fixed to a jig such as a tripod so that it can capture a predetermined range. For example, as shown in the upper part of Fig. 1, the stereo camera 10 is placed in front of a subject, a performer 1. Then, from the images captured by the stereo camera 10, a stereoscopic image 5 is generated and displayed on a stereoscopic display device 40.

[0036] At the filming location, the stereo camera 10 is oriented horizontally (Z direction) to film the performer 1 from the front. At the viewing location, the stereoscopic display device 40 is positioned so that the display surface 41 is perpendicular to the horizontal direction (Z direction). In other words, the display surface 41 is positioned vertically. This ensures that the depth direction of the stereo camera 10 coincides with the depth direction relative to the display surface 41.

[0037] In this embodiment, the scale of the three-dimensional image 6 displayed by the stereoscopic display device 40 is set to life-size. Therefore, the stereoscopic video 5 is configured so that the three-dimensional image 6 of the subject (such as the performer 1) is displayed at life-size. Here, the scale of the three-dimensional image 6 being life-size means that the scale of the space perceived by the viewer 2 as the three-dimensional image 6 is life-size. For example, assume that the three-dimensional image 6 of the subject is perceived at a position 1 m away from the viewer 2. If the scale of the three-dimensional image 6 is set to life-size, the size of the subject perceived as the three-dimensional image 6 will be the same as the size of the object seen by the viewer 2 when an object identical to the subject is actually placed 1 m away from the viewer 2 in the space where the viewer 2 is present.

[0038] Furthermore, a virtual display surface 11 is set at the shooting location. Here, the virtual display surface 11 is an assumed display surface 41 of the stereoscopic display device 40 in the shooting space of the stereo camera 10. In the upper diagram of FIG. 1 , the virtual display surface 11 is schematically illustrated by a dotted line. For example, when a stereoscopic image 6 is displayed by the stereoscopic display device 40 at the viewing location, the space perceived by the viewer 2 through the display surface 41 is the space seen through the virtual display surface 11 from the stereoscopic camera 10 at the shooting location. The virtual display surface 11 may be a virtually set surface, or may be a rectangular frame or the like that is actually provided at the shooting location.

[0039] When generating the stereoscopic video 5, the images captured by the stereo camera 10 are cropped and the angle of view is controlled, for example, depending on the size (display size) of the display surface 41 of the stereoscopic display device 40 and the relative positional relationship between the viewpoint of the viewer 2 and the display surface 41. Also, for example, the parallax of the stereoscopic video 5 is controlled so that the parallax is zero at the distance from the stereo camera 10 to the virtual display surface 11 (the viewing distance, which will be described later). The stereoscopic video 5 generated in this manner is displayed by the stereoscopic display device 40, thereby making it possible to realize a life-size stereoscopic display with a sense of seamlessness. The method of generating the stereoscopic video 5 will be described in detail later with reference to FIG. 13 etc.

[0040] 1, for example, a tea bowl 3 is placed in front of a performer 1 seated on the floor 9a of the filming location, and the movements of a traditional Japanese tea ceremony are demonstrated. In this case, both the performer 1 and the tea bowl 3 are photographed by a stereo camera 10. Here, the far end of the tea bowl 3 as seen from the performer 1 (the end on the stereo camera 10 side) overlaps with a virtual display surface 11. The height of the shooting viewpoint P of the stereo camera 10 from the floor 9a is 1 m, and the distance to the virtual display surface 11 is 1 m. The distance from the shooting viewpoint P to the performer 1 is 1.5 m.

[0041] At the viewing location, the stereoscopic image 5 generated using the above-described method from the images captured by the stereo camera 10 is displayed on the display surface 41 of the stereoscopic display device 40. Here, the height of the viewer 2's viewpoint Q from the floor 9b is 1 m, and the distance to the display surface 41 is 1 m. In this case, the viewer 2 viewing the stereoscopic display device 40 can perceive the tea bowl 3 and performer 1 at the shooting location as life-size stereoscopic images 6 through the display surface 41. For example, the performer 1 appears to be 1.5 m in front of the viewer 2. The viewer 2 also perceives the floor 9b of the viewing location as being directly connected to the floor 9a of the shooting location, which is displayed as the stereoscopic image 6, thereby realizing a stereoscopic display with a sense of continuity.

[0042] On the other hand, the appearance of the stereoscopic image 6 displayed by the stereoscopic display device 40 changes depending on the state (viewing conditions) of the viewer 2, such as the characteristics of each viewer 2 and the viewing position of the viewer 2. In other words, the relative positional relationship between the stereoscopic display device 40, which is referenced when generating the stereoscopic video 5, and the viewpoint of the viewer 2 changes when the state of the viewer 2 changes. As a result, for example, the size of the stereoscopic image 6 may deviate from the actual size, or the angle of the floor surface 9a may appear distorted. For this reason, in order to achieve the above-mentioned actual size stereoscopic display with a sense of continuity, it is important to control the imaging side in accordance with the state of the viewer 2.

[0043] Therefore, in this embodiment, viewer-side information (viewing parameters, described later) required to realize suitable stereoscopic display suited to the state of the viewer 2 is acquired using a sensing device (viewer sensor 30) such as an eye tracker. Furthermore, the viewer-side information is transmitted to the imaging side as meta-information, and the shooting viewpoint of the stereo camera 10 is mechanically controlled in accordance with this information. Specifically, the shooting viewpoint of the stereo camera 10 is adjusted using a mechanism such as a slider that mechanically moves the stereo camera 10. The images from the stereo camera 10 with the shooting viewpoint adjusted in this manner are processed to generate the stereoscopic image 5. This makes it possible to realize suitable stereoscopic display for everyone.

[0044] In this embodiment, the stereoscopic video 5 is an example of a stereoscopic image. In this disclosure, the term "image" includes both a moving image (video) that changes over time and a still image that does not change over time. While the following mainly describes a case where a video of the performer 1 is used, the present technology can also be applied to a case where a still image of the performer 1 is used.

[0045] [Configuration of 3D Telepresence System] Figure 2 is a block diagram showing an example configuration of a 3D telepresence system. The 3D telepresence system 100 has, as devices provided at the shooting base, an image capturing device 20, an image capturing controller 21, and a transmission device 22. The 3D telepresence system 100 also has, as devices provided at the viewer base, a viewer sensor 30, a viewer controller 31, and a stereoscopic display device 40.

[0046] First, the device installed at the base of the imaging side will be described. Fig. 3 is a schematic perspective view showing an example configuration of the imaging device 20. The imaging device 20 has the above-mentioned stereo camera 10 and an adjustment mechanism 12. The imaging device 20 is configured so that the imaging viewpoint P of the stereo camera 10 can be mechanically moved by driving the adjustment mechanism 12.

[0047] The stereo camera 10 is a twin camera with two cameras arranged on the left and right, including a right camera 10R and a left camera 10L. The right camera 10R is arranged on the right side of the stereo camera 10 and captures images to be displayed to the right eye of the viewer 2. The left camera 10L is arranged on the left side of the stereo camera 10 and captures images to be displayed to the left eye of the viewer 2. Any digital cameras equipped with image sensors such as a CMOS (Complementary Metal-Oxide Semiconductor) sensor or a CCD (Charge Coupled Device) sensor can be used as the right camera 10R and the left camera 10L.

[0048] In the following description, the center of the opening end of the right camera 10R through which external light enters will be referred to as the shooting viewpoint PR of the right camera 10R. Similarly, the center of the opening end of the left camera 10L through which external light enters will be referred to as the shooting viewpoint PL of the left camera 10L. In FIG. 2 , the shooting viewpoints PR and PL are schematically illustrated by black dots. Both the shooting viewpoints PR and PL become the shooting viewpoint P of the stereo camera 10.

[0049] Furthermore, the image captured by the right camera 10R will be referred to as the right camera image, and the image captured by the left camera 10L will be referred to as the left camera image. The right camera image and the left camera image may be collectively referred to as the stereo camera image. Of the stereoscopic image 5, the image for the right eye is generated by processing the right camera image, and the image for the left eye is generated by processing the left camera image.

[0050] 4 is a schematic diagram showing an example of a stereo camera image. The solid-line rectangular area represents the image range of the right camera image 4R and the left camera image 4L captured by the stereo camera 10. Here, the performer 1 and the tea bowl 3, which are the subjects of the stereo camera 10, are shown in the diagram.

[0051] The right camera 10R and the left camera 10L that make up the stereo camera 10 have different positions (shooting viewpoints PR and PL) in the left-right direction. Therefore, even if the same subject is shot in the right camera image 4R and the left camera image 4L, the position of the subject in each image will be shifted in the left-right direction depending on the depth. This shift in the left-right direction depending on the depth becomes parallax between the right camera image 4R and the left camera image 4L.

[0052] 3 , the adjustment mechanism 12 is a mechanism that mechanically moves the shooting viewpoint P of the stereo camera 10. In this embodiment, the adjustment mechanism 12 mechanically changes the inter-camera distance (ICD) of the stereo camera 10, the height position of the stereo camera 10, and the depth position of the stereo camera 10. The inter-camera distance, the height position, and the depth position of the stereo camera 10 are all adjustment parameters for adjusting the position of the shooting viewpoint P.

[0053] Fig. 5 is a schematic diagram showing adjustment parameters for the shooting viewpoint P of the stereo camera 10. The upper diagram of Fig. 5 shows the shooting site as seen from the X direction (height direction), and schematically illustrates the right camera 10R and left camera 10L of the stereo camera 10. The lower diagram of Fig. 5 shows the shooting site as seen from the Y direction (left-right direction), and corresponds to the upper diagram of Fig. 1.

[0054] The inter-camera distance ICD of the stereo camera 10 is the distance between the right camera 10R and the left camera 10L, and more specifically, the distance between the shooting viewpoint PR and the shooting viewpoint PL. The height position Ch of the stereo camera 10 is the position of the stereo camera 10 along the height direction (X direction) and is expressed, for example, as the height relative to the floor surface 9a of the shooting location. The depth position Cd of the stereo camera 10 is the position of the stereo camera 10 along the depth direction (Z direction) and is expressed, for example, as the distance relative to the virtual display surface 11.

[0055] 3 , in this embodiment, the stereo camera 10 is mechanically moved by driving the adjustment mechanism 12, and the position of the imaging viewpoint P (imaging viewpoint PR and imaging viewpoint PL) of the stereo camera 10 is adjusted. The adjustment mechanism 12 has a depth adjustment unit 13 a, a height adjustment unit 13 b, an inter-camera distance adjustment unit 13 c, and a control unit (not shown) that independently controls each adjustment unit. The control unit is connected to an imaging controller 21, which will be described later.

[0056] The depth adjustment unit 13a is a linear motion mechanism that moves the stereo camera 10 in the depth direction. The depth adjustment unit 13a has a support unit 14a that extends along a movement axis, and a movable unit 15a that is supported by the support unit 14a and moves linearly along the movement axis. The support unit 14a is fixed to a jig such as a tripod so that the movement axis of the movable unit 15a is parallel to the depth direction (Z direction).

[0057] The height adjustment unit 13b is a linear motion mechanism that moves the stereo camera 10 in the height direction. The height adjustment unit 13a has a support unit 14b that extends along a movement axis and a movable unit 15b that is supported by the support unit 14b and moves linearly along the movement axis. The support unit 14b is fixed to the movable unit 15a of the depth adjustment unit 13a so that the movement axis of the movable unit 15b is parallel to the height direction (X direction).

[0058] The camera distance adjustment unit 13c is a linear motion mechanism that changes the inter-camera distance between the right camera 10R and the left camera 10L. The right camera 10R and the left camera 10L are attached side by side to the camera distance adjustment unit 13c along its movement axis. The camera distance adjustment unit 13c moves both or one of the right camera 10R and the left camera 10L along the movement axis. The camera distance adjustment unit 13c is also fixed to the movable part 15b of the height adjustment unit 13b so that the movement axis of each camera is parallel to the left-right direction (Y direction).

[0059] The right camera 10R and the left camera 10L are arranged so that their shooting directions (optical axis directions of the lenses) are parallel to each other and perpendicular to the movement axis of the inter-camera distance adjustment unit 13c. Therefore, the shooting directions of the right camera 10R and the left camera 10L are both parallel to the depth direction (Z direction).

[0060] For example, by driving the inter-camera distance adjustment unit 13c, the right camera 10R or the left camera 10L is moved in the left-right direction (Y direction). This allows the adjustment mechanism 12 to adjust the inter-camera distance ICD of the stereo camera 10. Furthermore, by driving the movable part 15b of the height adjustment unit 13b, the stereo camera 10 is moved in the height direction (X direction) together with the inter-camera distance adjustment unit 13c fixed to the movable part 15b. This allows the adjustment mechanism 12 to adjust the height position Ch of the stereo camera 10. Furthermore, by driving the movable part 15a of the depth adjustment unit 13a, the stereo camera 10 is moved in the depth direction (Z direction) together with the height adjustment unit 13b and the inter-camera distance adjustment unit 13c fixed to the movable part 15a. This allows the adjustment mechanism 12 to adjust the depth position Cd of the stereo camera 10.

[0061] The configurations of the depth adjustment unit 13a, the height adjustment unit 13b, and the inter-camera distance adjustment unit 13c are not limited. For example, a slider in which a wire or a ball screw is driven by a motor may be used. Alternatively, a movement mechanism in which a motor is provided in a movable part may be used. Furthermore, the depth adjustment unit 13a and the height adjustment unit 13b may move the jig (such as a tripod) itself that fixes the stereo camera 10, or may be configured as part of the jig.

[0062] The adjustment mechanism 12 may also be provided with a position sensor (not shown) that detects the position and amount of movement of each movement mechanism (depth adjustment unit 13 a, height adjustment unit 13 b, and inter-camera distance adjustment unit 13 c). In this case, the control unit drives each movement mechanism by referring to the output of the position sensor. This makes it possible to adjust the shooting viewpoint P of the stereo camera 10 with high precision.

[0063] 2 , the imaging-side controller 21 is a device that controls the imaging device 20 (adjustment mechanism 12 and stereo camera 10) and performs processing to generate the stereoscopic video 5 from the video of the stereo camera 10, and is configured using, for example, a PC (Personal Computer), etc. The imaging-side controller 21 has a storage unit 23 and a control device 24.

[0064] The storage unit 23 is a non-volatile storage device, such as a solid state drive (SSD) or a hard disk drive (HDD). A control program is stored in the storage unit 23. The control program is, for example, a program that controls the overall operation of the imaging controller 21. In this embodiment, the storage unit 23 corresponds to a computer-readable recording medium on which a program is recorded. The control program corresponds to the program recorded on the recording medium.

[0065] The control device 24 is a computing device that controls the operation of the imaging controller 21. The control device 24 has hardware components necessary for a computer, such as a CPU and memory (RAM, ROM). The CPU loads a control program stored in the storage unit 23 into the RAM and executes it, thereby performing various processes. In this embodiment, the control device 24 corresponds to an information processing device.

[0066] The control device 24 may be a programmable logic device (PLD) such as a field programmable gate array (FPGA), or another device such as an application specific integrated circuit (ASIC). Alternatively, the control device 24 may be a graphics processing unit (GPU).

[0067] In this embodiment, the CPU of the control device 24 executes a program (control program) according to this embodiment, thereby realizing a camera control unit 25 and an image processing unit 26 as functional blocks. These functional blocks then execute the information processing method according to this embodiment. Note that dedicated hardware such as an IC (integrated circuit) may be used as appropriate to realize each functional block.

[0068] The camera control unit 25 controls the adjustment mechanism 12 that mechanically moves the stereo camera 10. In this embodiment, viewing parameters are acquired by the camera control unit 25. Then, the stereo camera 10 is mechanically controlled to adjust the shooting viewpoint P (PR and PL) of the stereo camera 10 based on the viewing parameters.

[0069] Fig. 6 is a schematic diagram showing the viewing parameters of viewer 2. The upper diagram of Fig. 6 shows the viewer's location as viewed from the X direction (height direction), and the right eye 7R and left eye 7L of viewer 2 are schematically illustrated. The lower diagram of Fig. 6 shows the viewer's location as viewed from the Y direction (left-right direction), and corresponds to the lower diagram of Fig. 1. Hereinafter, the position of viewer 2's right eye 7R will be referred to as right-eye viewpoint QR, and the position of viewer 2's left eye 7L will be referred to as left-eye viewpoint QL. Both right-eye viewpoint QR and left-eye viewpoint QL are viewer 2's viewpoint Q.

[0070] The viewing parameters are parameters that represent the state of the viewer 2, and vary depending on the characteristics and viewing manner of each viewer 2. The viewing parameters can also be considered as viewer information. In this embodiment, the viewing parameters include the inter-pupillary distance (IPD) of the viewer 2, the viewing height Vh of the viewer 2, and the viewing distance Vd of the viewer 2.

[0071] The interpupillary distance IPD is the distance between the pupil of the viewer's right eye 7R and the pupil of the viewer's left eye 7L. Therefore, IPD can also be said to be the distance between the right eye viewpoint QR and the left eye viewpoint QL. The viewing height Vh is a parameter representing the height of the position from which the viewer 2 views the stereoscopic display device 40. For example, it is the height of the viewer's 2 viewpoint Q (QR and QL) from the floor surface 9b. Furthermore, if the viewer 2's posture when viewing the stereoscopic display device 40 is fixed (for example, a case in which the viewer 2 views while sitting on the floor surface 9b, a case in which the viewer 2 views while sitting in a chair, a case in which the viewer 2 views while standing, etc.), the height of the viewer 2 may be acquired as the viewing height Vh. The viewing distance Vd of the viewer 2 is a parameter representing the distance between the viewer 2 and the stereoscopic display device 40, and is, for example, the distance between the viewer's 2 viewpoint Q (QR and QL) and the display surface 41.

[0072] In this embodiment, these viewing parameters are detected by a sensor (a viewer sensor 30, described later) that measures the viewer 2. This makes it possible to monitor the viewing parameters in real time, for example, and even if the position or posture of the viewer 2 changes, it becomes possible to detect and provide feedback on the appropriate viewing parameters.

[0073] The camera control unit 25 controls the adjustment mechanism 12 based on the viewing parameters. For example, the camera control unit 25 generates control commands for driving each part of the adjustment mechanism 12 based on the viewing parameters. By driving the adjustment mechanism 12 in accordance with these control command values, the shooting viewpoint P of the stereo camera 10 is adjusted. In this way, the camera control unit 25 mechanically controls the stereo camera 10 by generating and outputting control commands (commands to the adjustment mechanism 12) for the adjustment mechanism 12.

[0074] Specifically, the camera control unit 25 controls the adjustment mechanism 12 to adjust the inter-camera distance ICD based on the inter-pupillary distance IPD of the viewer 2. For example, the camera control unit 25 calculates a control value for the inter-camera distance ICD according to the inter-pupillary distance IPD and outputs it to the control unit of the adjustment mechanism 12. The control unit drives the inter-camera distance adjustment unit 13c, and moves both or one of the right camera 10R and the left camera 10L in the left-right direction so as to achieve the inter-camera distance ICD indicated by the control value.

[0075] Furthermore, the camera control unit 25 controls the adjustment mechanism 12 to adjust the height position Ch of the stereo camera 10 based on the viewing height Vh of the viewer 2. For example, the camera control unit 25 calculates the height position Ch of the stereo camera 10 according to the viewing height Vh and outputs it to the control unit of the adjustment mechanism 12. The control unit drives the height adjustment unit 13b, and moves the entire stereo camera 10 in the height direction so as to achieve the height position Ch indicated by the control value.

[0076] Furthermore, the camera control unit 25 controls the adjustment mechanism 12 to adjust the depth position Cd of the stereo camera 10 based on the viewing distance Vd of the viewer 2. For example, the camera control unit 25 calculates the depth position Cd of the stereo camera 10 according to the viewing distance Vd and outputs it to the control unit of the adjustment mechanism 12. The control unit drives the depth adjustment unit 13a, and moves the entire stereo camera 10 in the depth direction so as to achieve the depth position Cd indicated by the control value.

[0077] The image processing unit 26 generates in real time a stereoscopic image 5 (image for the right eye and image for the left eye) to be displayed on the stereoscopic display device 40 from the images (right camera image 4R and left camera image 4L) captured by the stereo camera 10 with the adjusted shooting viewpoint P. This makes it possible to generate in real time a stereoscopic image 5 that matches the state (viewing parameters) of the viewer 2. In this embodiment, the image processing unit 26 corresponds to an image generation unit.

[0078] For example, when stereoscopic display is performed using images from the stereo camera 10, the scale and depth (amount of protrusion or recession) of the stereoscopic image 6 can be controlled by the stereoscopic image 5 displayed on the stereoscopic display device 40. Conversely, when the scale and depth of the stereoscopic image 6 are fixed, it is necessary to generate the stereoscopic image 5 by appropriately processing the images captured by the stereo camera 10.

[0079] Therefore, the image processing unit 26 performs a shift process to adjust the parallax on the images (right camera image 4R and left camera image 4L) captured by the stereo camera 10, a trimming process to adjust the angle of view to the display range of the stereoscopic display device 40, and an enlargement process to fit the image to a predetermined image size, thereby generating a stereoscopic image 5.

[0080] The shifting process is a process of shifting the images captured by the stereo camera 10 to adjust the parallax. The shifting process makes it possible to adjust the depth of the stereoscopic image 6. The trimming process is a process of cutting out the images captured by the stereo camera 10 and adjusting the angle of view to the display range of the stereoscopic display device 40. The enlargement process is a process of enlarging the trimmed images to a predetermined image size. Here, the predetermined image size is, for example, the size (resolution) of the image used for transmitting and displaying the stereoscopic image 5. The trimming and enlargement processes make it possible to display the stereoscopic image 6 at an appropriate scale.

[0081] Fig. 7 is a schematic diagram showing an example of a stereoscopic image 5. The solid-line rectangular area represents the image range of the right-eye image 5R and the left-eye image 5L that make up the stereoscopic image 5. The right-eye image 5R and the left-eye image 5L shown in Fig. 7 are images generated from the right camera image 4R and the left camera image 4L shown in Fig. 4.

[0082] For example, the right-eye image 5R (left-eye image 5L) is an image obtained by shifting the right camera image 4R (left camera image 4L) to obtain a desired depth, and then cropping and enlarging the shifted image to fit the display range of the stereoscopic display device 40. Specific details of the shifting process, trimming process, and enlarging process will be described in detail later with reference to FIG. 13 .

[0083] The image processing unit 26 also performs 3D processing to display the stereoscopic image 5 as a stereoscopic image 6. The 3D processing includes, for example, distortion correction processing and parallelization processing. In the distortion correction processing, image distortion according to the characteristics of the optical systems of the right camera 10R and the left camera 10L is corrected. In the parallelization processing, the right eye image 5R and the left eye image 5L are stereo-parallelized. In addition, the specific content of the 3D processing is not limited, and any image processing required for stereoscopic display may be performed.

[0084] 2 , the transmission device 22 transmits the stereoscopic video 5 generated by the video processing unit 26 to a viewing-side controller 31 provided at the viewing site via a predetermined network. A PC or the like is used as the transmission device 22. Note that it is not always necessary to provide the transmission device 22; for example, the shooting-side controller 21 and the viewing-side controller 31 may be directly connected.

[0085] The processing by the image processing unit 26 may be performed after the transmission device 22. For example, the images captured by the stereo camera 10 (right camera image 4R and left camera image 4L) may be transmitted by the transmission device 22 to the viewing-side controller 31. The viewing-side controller 31 may also process the transmitted images to generate the stereoscopic image 5 (right eye image 5R and left eye image 5L).

[0086] Next, a device installed at the viewing site will be described. The stereoscopic display device 40 is a device that displays a right-eye image 5R and a left-eye image 5L, which are stereoscopic images 5, independently to the right eye 7R and left eye 7L of the viewer 2, thereby allowing the viewer 2 to perceive a stereoscopic image 6 (3D image) utilizing parallax. The stereoscopic display device 40 displays the stereoscopic image 5 captured by the stereo camera 10 in real time.

[0087] In this embodiment, a naked-eye stereoscopic display device 40 is used, which allows the viewer 2 to view a stereoscopic image 6 without using special glasses or the like. The naked-eye stereoscopic display device 40 includes, for example, a display panel that displays images and an optical element that controls the direction of light emitted from each pixel of the display panel. The display panel may be, for example, a liquid crystal display (LCD), a plasma display panel (PDP), or an organic electroluminescence (EL) panel. The optical element may be, for example, a lenticular lens or a parallax barrier.

[0088] An eye-tracking device may also be used as the stereoscopic display device 40. In this case, the viewpoint Q (right eye viewpoint QR and left eye viewpoint QL) of the viewer 2 is detected, and pixels for displaying the right eye image 5R and the left eye image 5L are allocated according to the viewpoint Q of the viewer 2. This makes it possible, for example, to allocate all pixels of the display panel to either the right eye image 5R or the left eye image 5L, thereby realizing high-resolution stereoscopic display.

[0089] The stereoscopic display device 40 does not have to be a naked-eye type. For example, a polarization type that displays the stereoscopic image 5 using polarized glasses or the like, or a frame sequential type that switches between the left and right stereoscopic images 5 for each frame using liquid crystal glasses or the like, may be used. In addition, the type of the stereoscopic display device 40 is not limited, and any display type that can individually display the right-eye image 5R and the left-eye image 5L to the right eye and the left eye of the viewer 2 may be used. For example, a stacked type in which display panels are stacked, a projector array type, or the like may be used.

[0090] Fig. 8 is a schematic diagram illustrating stereoscopic display by a stereoscopic display device 40. A right-eye image 5R and a left-eye image 5L are displayed on a display surface 41 of the stereoscopic display device 40 shown in Fig. 8. The right-eye image 5R is displayed toward the right eye 7R of the viewer 2, and the left-eye image 5L is displayed toward the left eye 7L of the viewer 2.

[0091] 8 also shows a point OR included in the right-eye image 5R and a point OL included in the left-eye image 5L. The points OR and OL are the same point O on the subject shown in each image. The distance between the points OR and OL on the display surface 41 is the amount of parallax for the point O.

[0092] In the left diagram of Fig. 8 , point OR is displayed on the right side of the display surface 41, and point OL is displayed on the left side. In this case, the right eye's line of sight from the right eye 7R toward point OR and the left eye's line of sight from the left eye 7L toward point OL intersect on the back side of the display surface 41. To the viewer 2, point O appears to be present at this intersection. Therefore, on the left side of Fig. 8 , point O appears to be recessed toward the back side of the display surface 41. In this way, the amount of parallax is considered negative when point OR in the right eye image 5R is to the right of point OL in the left eye image 5L.

[0093] In the diagram on the right side of Fig. 8, point OR is displayed on the left side and point OL is displayed on the right side on the display surface 41. In this case, the right eye's line of sight from the right eye 7R toward point OR and the left eye's line of sight from the left eye 7L toward point OL intersect on the near side of the display surface 41, so point O appears to protrude in front of the display surface 41. In this way, the amount of parallax is considered positive when point OR in the right eye image 5R is to the left of point OL in the left eye image 5L.

[0094] When points OR and OL overlap on the display surface 41, that is, when the parallax amount is 0, point O is perceived as a point on the display surface 41. In this way, the depth of point O on the subject is determined by the parallax between the right-eye image 5R and the left-eye image 5L. For example, the shift process performed by the image processing unit 26 described above adjusts the parallax on the display surface 41 to adjust the depth of the stereoscopic image 6.

[0095] Returning to Fig. 2 , the viewer sensor 30 is a sensor for detecting viewing parameters from the viewer 2 of the stereoscopic display device 40. The viewer sensor 30 is, for example, an eye-tracking sensing device provided in the stereoscopic display device 40, and a sensor capable of detecting the positions of the right eye 7R and left eye 7L of the viewer 2 is used. For example, a stereo camera or a TOF (Time of Flight) camera is used as the viewer sensor 30. Alternatively, a dedicated device for detecting viewing parameters may be provided as the viewer sensor 30. The detection results by the viewer sensor 30 (such as the stereo image and ToF data of the viewer 2) are output to the viewer-side controller 31.

[0096] The viewer-side controller 31 is a device that performs processes such as calculating viewing parameters and displaying stereoscopic video 5 on the stereoscopic display device 40, and is configured using, for example, a PC (Personal Computer). The viewer-side controller 31 has, as functional blocks, a stereoscopic display processing unit 32 and a viewer information measurement unit 33.

[0097] The stereoscopic display processing unit 32 receives the stereoscopic video 5 transmitted from the transmission device 22 at the shooting site and executes processing to display the stereoscopic video 5 so that it can be viewed as a stereoscopic image 6. This processing involves, for example, allocating pixels for displaying the right-eye video 5R and the left-eye video 5L on the display panel of the stereoscopic display device 40. For example, in an eye-tracking device, pixels for displaying each video are allocated based on the detection results of the viewer sensor 30.

[0098] The viewer information measurement unit 33 measures the viewing parameters (pupillary distance IPD of the viewer 2, viewing height Vh of the viewer 2, and viewing distance Vd of the viewer 2) described with reference to Fig. 6 based on the detection results of the viewer sensor 30. For example, the three-dimensional positions of the pupil centers of the right eye 7R and left eye 7L of the viewer 2 (right eye viewpoint QR and left eye viewpoint QL) are detected from stereo images or ToF data obtained by capturing an image of the viewer 2. In this case, the distance between the right eye viewpoint QR and the left eye viewpoint QL is calculated as the IPD.

[0099] The height positions of the right eye viewpoint QR and the left eye viewpoint QL are calculated as the viewing height Vh. Note that the viewing height Vh may be the average value of the height positions of QR and QL, or the height position of the midpoint between QR and QL. Alternatively, the height of the viewer 2 may be detected, and the viewing height Vh may be calculated from the height value and the posture of the viewer 2.

[0100] Furthermore, based on the depth positions of the right eye viewpoint QR and the left eye viewpoint QL, the distance from the display surface 41 of the stereoscopic display device 40 to each of the viewpoints QR and QL is calculated as a viewing distance Vd. The value used to calculate the viewing distance Vd may be the average value of the depth positions of QR and QL, or the depth position of the midpoint between QR and QL.

[0101] Besides, the method of calculating each of the viewing parameters, i.e., interpupillary distance IPD, viewing height Vh, and viewing distance Vd, is not limited to the above. The viewing parameters calculated by the viewer information measurement unit 33 are transmitted to the imaging-side controller 21 (camera control unit 25) at the viewer's base via a predetermined network.

[0102] [Operation of 3D Telepresence System] Fig. 9 is a flowchart showing an example of the operation of the 3D telepresence system 100. The process shown in Fig. 9 is, for example, a loop process that is repeatedly executed while the 3D telepresence system 100 is in use.

[0103] First, at the viewing location, the viewer 2 viewing the stereoscopic display device 40 is detected (step 101). For example, when the viewer 2 moves in front of the stereoscopic display device 40 and starts viewing the stereoscopic image 6, detection of the viewer 2 using the viewer sensor 30 begins. Next, the viewer information measurement unit 33 measures viewing parameters based on the detection results of the viewer sensor 30 (step 102). The measured viewing parameters (interpupillary distance IPD, viewing height Vh, and viewing distance Vd) are sequentially transmitted to the filming location.

[0104] At the shooting location, the camera control unit 25 calculates control values ​​for the stereo camera 10 based on the viewing parameters (step 103). The control values ​​for the stereo camera 10 are adjustment values ​​for adjusting the position of the shooting viewpoint P of the stereo camera 10 (inter-camera distance ICD, height position Ch, depth position Cd). The calculated control values ​​are transmitted to the adjustment mechanism 12 (step 104), and the adjustment mechanism 12 performs mechanical control of the stereo camera 10 (step 105).

[0105] 1 , the 3D telepresence system 100 provides a life-size, seamless stereoscopic display. For this reason, in this embodiment, the adjustment mechanism 12 is controlled so that the positional relationship between the virtual display surface 11, which is an image of the display surface 41 of the stereoscopic display device 40 in the shooting space of the stereo camera 10, and the shooting viewpoint P of the stereo camera 10 is the same as the positional relationship between the display surface 41 of the stereoscopic display device 40 and the viewpoint Q of the viewer 2.

[0106] 10 is a schematic diagram showing an example of adjustment of the inter-camera distance ICD of the stereo camera 10. In this embodiment, the camera control unit 25 matches the inter-camera distance ICD of the stereo camera 10 with the inter-pupillary distance IPD of the viewer 2. That is, the control value of the inter-camera distance adjustment unit 13c of the adjustment mechanism 12 is calculated so that ICD = IPD.

[0107] For example, as shown in the upper part of Fig. 10, if the inter-camera distance ICD before adjustment is shorter than the interpupillary distance IPD of the viewer 2, the distance between the right camera 10R and the left camera 10L is increased until ICD matches the IPD, as shown in the lower part of Fig. 10. For example, if both the right camera 10R and the left camera 10L are movable, the right camera 10R is moved to the right and the left camera 10L is moved to the left. Also, if one of the right camera 10R and the left camera 10L is movable, the movable camera is moved away from the other camera. Conversely, if the inter-camera distance ICD before adjustment is longer than the IPD of the viewer 2, the distance between the right camera 10R and the left camera 10L is decreased until ICD matches the IPD.

[0108] 11 is a schematic diagram showing an example of adjusting the height position Ch of the stereo camera 10. In this embodiment, the camera control unit 25 matches the height position Ch of the stereo camera 10 with the viewing height Vh of the viewer 2. That is, the control value of the height adjustment unit 13b of the adjustment mechanism 12 is calculated so that Ch = Vh.

[0109] 11 , if the pre-adjustment height position Ch is lower than the viewing height Vh of the viewer 2, the stereo camera 10 is moved upward until Ch coincides with Vh. Conversely, if the pre-adjustment height position Ch is higher than the viewing height Vh of the viewer 2, the stereo camera 10 is moved downward until Ch coincides with Vh. Note that the height of the viewer 2 can also be used as the viewing height Vh. For example, a reference height and a reference position in the height direction of the stereo camera 10 corresponding to that height are set. In this case, the stereo camera 10 is moved appropriately in the vertical direction from the reference position, with the difference between the reference height and the height of the viewer 2 being used as the movement amount.

[0110] 12 is a schematic diagram showing an example of adjusting the depth position Cd of the stereo camera 10. In this embodiment, the camera control unit 25 matches the distance between the stereo camera 10 and the virtual display surface 11 with the viewing distance Vd of the viewer 2. Here, the depth position Cd is the distance from the virtual display surface 11 to the shooting viewpoint P of the stereo camera 10. In other words, the origin of the depth position Cd is set to the virtual display surface 11. In this case, the control value of the depth adjustment unit 13a of the adjustment mechanism 12 is calculated so that Cd=Vd.

[0111] 12, if the depth position Cd before adjustment is shorter than the viewing distance Vd of the viewer 2, the stereo camera 10 is moved in a direction away from the virtual display surface 11 (to the left in the figure) until Cd matches Vd. Conversely, if the depth position Cd before adjustment is longer than the viewing distance Vd of the viewer 2, the stereo camera 10 is moved in a direction closer to the virtual display surface 11 (to the right in the figure) until Cd matches Vd.

[0112] 10, 11, and 12 can be said to be control that reproduces the viewpoint Q of viewer Q viewing display surface 41 at the viewing location as the shooting viewpoint P of stereo camera 10 at the shooting location. By controlling the shooting viewpoint P in this way, it is possible to generate appropriate stereoscopic video 5 that is suited to the state of viewer 2, for example.

[0113] 9 , when the shooting viewpoint P is mechanically controlled, the image processing unit 26 processes the images from the stereo camera 10 (right camera image 4R and left camera image 4L) to generate a stereoscopic image 5 (right eye image 5R and left eye image 5L) (step 106). Here, shifting, trimming, enlarging, etc. are performed on the images from the stereo camera 10 so that a life-size stereoscopic display with a sense of seamlessness is possible, and the stereoscopic image 5 is generated.

[0114] The generated stereoscopic video 5 is output to the transmission device 22. The transmission device 22 transmits the stereoscopic video 5 to the viewing location via a predetermined network (step 107). At the viewing location, the stereoscopic display processing unit 32 of the viewing controller 31 performs display processing (pixel allocation, etc.) of the stereoscopic video 5. Once the display processing is complete, the stereoscopic video 5 is displayed by the stereoscopic display device 40 (step 108). This allows the viewer 2 of the stereoscopic display device 40 to perceive a life-size, seamless stereoscopic image 6 via the display surface 41. The operation of the video processing unit 26 will be described in detail below.

[0115] [Video Processing] Fig. 13 is a schematic diagram showing an example of video processing. Fig. 13 shows a flow of video processing performed by the video processing unit 26. This processing is image processing performed for each image constituting a video frame.

[0116] In the following description, it is assumed that the stereoscopic video 5 is generated at the same resolution as the stereo camera 10 (right camera 10R and left camera 10L). Therefore, the image size of the ultimately generated stereoscopic video 5 will be the size indicated by the resolution of the stereo camera 10. In the example shown in FIG. 13, the number of pixels in the vertical direction is 3840 pixels, and the number of pixels in the horizontal direction is 2160 pixels. Note that the image size of the stereoscopic video 5 does not necessarily have to match the image size of the stereoscopic camera 10. For example, any image size may be set depending on the video transmission speed, the resolution of the stereoscopic display device 40, etc.

[0117] Here, we will explain the process of processing the left camera image 4L captured by the left camera 10L of the stereo camera 10 to generate a left-eye image 5L. On the left side of Fig. 13, the left camera image 4L is schematically illustrated by a dotted rectangular area. Note that the solid rectangular area illustrated within the left camera image 4L is the area of ​​a virtual display surface 11 that is assumed to be the display surface 41 of the stereoscopic display device 40. The virtual display surface 11 is not, for example, an object that is actually captured, and is not captured in the left camera image 4L.

[0118] First, the image processing unit 26 shifts the images captured by the stereo camera 10 so that the parallax in the depth direction of the virtual display surface 11 becomes zero. This process corresponds to a shift process for adjusting the parallax. Here, the shooting viewpoint P of the stereo camera 10 is located 1 m away from the virtual display surface 11 (Cd = 1 m). In this case, the entire image is shifted so that the parallax of point O on the subject, which is 1 m behind the shooting viewpoint P (to the right in the figure), becomes zero.

[0119] The center of Figure 13 shows a schematic diagram of the left camera image 4L after being subjected to shift processing. In this example, the camera-side edge of the bowl 3 is point O, where the depth coincides with that of the virtual display surface 11, so the left camera image 4L is shifted so that the parallax becomes zero at the camera-side edge of the bowl 3. In Figure 13, the left camera image 4L is shifted to the right relative to the image frame (a frame of 3840 pixels x 2160 pixels).

[0120] In this way, by setting the parallax in the depth direction of the virtual display surface 11 to zero, the viewer 2 perceives point O as a point on the display surface 41 of the stereoscopic display device 40. This makes it possible to match the depth as seen from the shooting viewpoint P of the stereo camera 10 with the depth as seen from the viewpoint Q of the viewer 2, for example.

[0121] Note that shifting the image creates areas within the image frame where no captured image exists. For example, in the central diagram of Figure 13, the area on the left side of the image frame is an area with no pixel values. Such areas are interpolated with an arbitrary pixel value, such as black.

[0122] When the shifting process is completed, the image processing unit 26 trims the area corresponding to the virtual display surface 11 as seen from the adjusted shooting viewpoint P from the image shifted by the shifting process. This process corresponds to a trimming process for adjusting the angle of view to the display range of the stereoscopic display device 40. For example, in the left camera image 4L shown in the center of Fig. 13, the area included in the solid-line rectangular range (virtual display surface 11) is cut out.

[0123] Once the trimming process is complete, the image processing unit 26 enlarges the trimmed image to a predetermined image size. This process corresponds to an enlargement process for adjusting an image to a predetermined image size. As described above, in this embodiment, the stereoscopic image 5 is generated at the resolution of the stereo camera 10, so the trimmed image is enlarged to this resolution. Therefore, the resolution of the stereo camera 10 corresponds to the predetermined image size. The right side of FIG. 13 shows an image obtained by enlarging the trimmed left camera image 4L to the resolution of the stereo camera 10 (3840 pixels x 2160 pixels). This image is used as the left-eye image 5L.

[0124] The right camera image 4R captured by the right camera 10R is also subjected to similar conversion processes (shifting, trimming, and enlargement) to generate the right-eye image 5R. Note that, in the shifting process, only the left camera image 4L may be shifted. Alternatively, both the left camera image 4L and the right camera image 4R may be shifted by half.

[0125] As described above, in this embodiment, the images captured by the stereo camera 10 are processed in accordance with the relative positions of the stereoscopic display device 40 (virtual display surface 11) and the stereo camera 10, and the display size of the stereoscopic display device 40 (the size of the virtual display surface 11). This makes it possible to accurately display, for example, a life-size stereoscopic image 6 with a sense of continuity.

[0126] As described above, the control device 24 of the imaging controller 21 according to this embodiment mechanically controls the stereo camera 10 that captures the stereoscopic video 5 based on viewing parameters such as the interpupillary distance IPD, viewing height Vh, and viewing distance Vd of the viewer 2 viewing the stereoscopic video 5. This makes it possible to adjust the imaging viewpoint P of the stereo camera 10 according to the state of the viewer, thereby realizing appropriate stereoscopic display that matches the state of the viewer 2.

[0127] When performing life-size or seamless stereoscopic display, the images from the stereo camera 10 are processed by image processing so that the viewer 2 perceives the stereoscopic image 6 in an appropriate manner. However, since the appearance of the stereoscopic image 6 varies depending on the characteristics of the individual viewer 2 and the viewing conditions, it is possible that the stereoscopic image 6 will not be perceived appropriately. In such cases, it is difficult to sufficiently correct the stereoscopic image 6 using image processing alone, making it difficult to provide a suitable stereoscopic display for everyone.

[0128] In this embodiment, information about the viewer 2 (viewing parameters) is sensed at the timing when the viewer 2 starts viewing the stereoscopic display device 40. The viewing parameters are transmitted to the imaging system, and the stereo camera 10 is mechanically controlled in accordance with the value of the viewing parameters, and the imaging viewpoint P of the stereo camera 10 is adjusted. This makes it possible to move the imaging viewpoint P appropriately and capture images that cannot be obtained by image processing.

[0129] As a result, it becomes possible to capture an image from a shooting viewpoint P that is suited to the state of the viewer 2 (interpupillary distance IPD, viewing height Vh, viewing distance Vd, etc.), and it becomes possible to adapt the appearance of the stereoscopic image 6 to the viewer 2. This makes it possible to provide a stereoscopic display that is suitable for everyone. It also makes it possible to accurately realize the stereoscopic display intended by the shooter, such as a life-size image or a stereoscopic display with a sense of seamlessness.

[0130] Second Embodiment A 3D telepresence system according to a second embodiment of the present technology will be described. In the following description, descriptions of parts having the same configuration and operation as those of the 3D telepresence system 100 described in the above embodiment will be omitted or simplified.

[0131] 14 is a schematic diagram showing an example configuration of a 3D telepresence system according to the second embodiment. The 3D telepresence system 200 according to this embodiment is configured to feed back the output of a position sensor 16 provided in the adjustment mechanism 12 to the processing of images captured by the stereo camera 10. The 3D telepresence system 200 has a configuration in which a difference measurement unit 27 is added to the 3D telepresence system 100 described with reference to FIG. 2, for example.

[0132] The position sensor 16 is a sensor that detects the position of each movement mechanism (depth adjustment unit 13 a, height adjustment unit 13 b, and inter-camera distance adjustment unit 13 c) provided in the adjustment mechanism 12. The position sensor 16 may also be a sensor that detects the amount of movement of each movement mechanism. There are no limitations on the type of position sensor 16, and for example, a sensor whose resistance value changes in response to the movement of a slider, a distance measurement sensor that uses a laser, etc. may be used. Note that the value detected by the position sensor 16 may not necessarily be the same as the specified adjustment value due to errors in mechanical control, etc.

[0133] For example, the position sensor 16 detects the position of the depth adjustment unit 13a, thereby detecting the depth position Cd' of the adjusted shooting viewpoint P. Furthermore, the position sensor 16 detects the position of the height adjustment unit 13b, thereby detecting the height position Ch' of the adjusted shooting viewpoint P. Furthermore, the position sensor 16 detects the position of the inter-camera distance adjustment unit 13c, thereby detecting the adjusted inter-camera distance ICD'. In this way, the position sensor 16 can also be said to be a sensor that detects the position of the shooting viewpoint P of the stereo camera 10.

[0134] The difference measurement unit 27 is a functional block configured by the control device 24 of the viewing-side controller 31, and calculates the difference between an adjustment value calculated from the viewing parameters and an actual mechanically controlled value. In other words, the difference measurement unit 27 calculates the difference between the adjustment value of the shooting viewpoint P according to the viewing parameters and the detection value of the position sensor 16.

[0135] The adjustment value of the shooting viewpoint P according to the viewing parameters is an adjustment value (control command value) calculated from the viewing parameters in order to control the adjustment mechanism. Therefore, the adjustment value of the shooting viewpoint P is, for example, the depth position Cd, the height position Ch, and the inter-camera distance ICD. Furthermore, the detection value of the position sensor 16 is, for example, the depth position Cd', the height position Ch', and the inter-camera distance ICD' detected using the position sensor 16. These values ​​are actual measurement values ​​that represent the actual position of the shooting viewpoint P. The difference measurement unit 27 calculates, for example, the difference in depth position (Cd-Cd'), the difference in height position (Ch-Ch'), and the difference in inter-camera distance (ICD-ICD').

[0136] The image processing unit 26 adjusts the images captured by the stereo camera 10 (the right camera image 4R and the left camera image 4L) based on the difference calculated by the difference measurement unit 27. For example, if the adjustment value Ch of the height position of the shooting viewpoint P is higher than the actual measurement value Ch', the image may be shifted upward by the difference in height position, and if the adjustment value Ch is lower than the actual measurement value Ch', the image may be shifted downward by the difference in height position. Furthermore, for example, if the adjustment value Cd of the depth position of the shooting viewpoint P is greater than the actual measurement value Cd', the amount of parallax of the image may be reduced, and if the adjustment value Cd is smaller than the actual measurement value Cd', the amount of parallax of the image may be increased. Furthermore, for example, if the adjustment value ICD of the inter-camera distance is greater than the actual measurement value ICD', the amount of parallax of the image may be reduced, and if the adjustment value ICD is smaller than the actual measurement value ICD', the amount of parallax of the image may be increased.

[0137] After adjusting the images captured by the stereo camera 10 using the calculation results of the difference measurement unit 27, the image processing described with reference to Fig. 13 etc. is performed. As described above, in this embodiment, in addition to mechanical control of the stereo camera 10, signal processing such as image shifting is added according to the value obtained by the difference measurement, making it possible to compensate for the accuracy that is lacking due to mechanical control. This makes it possible to improve the accuracy of the stereoscopic image 5, and to realize a highly accurate stereoscopic display.

[0138] 15 is a schematic diagram showing an example of the configuration of a 3D telepresence system according to a third embodiment. In a 3D telepresence system 300 according to this embodiment, the depth of a stereoscopic image 6 is adjusted using image processing instead of adjusting the depth of a stereo camera 10. For this reason, the 3D telepresence system 300 does not include a depth adjustment mechanism (depth adjustment unit 13 a) in the adjustment mechanism 12.

[0139] In this embodiment, the image processing unit 26 shifts the images captured by the stereo camera 10 (the right camera image 4R and the left camera image 4L) so that the parallax in the depth direction of the virtual display surface 11 corresponds to the viewing distance Vd of the viewer 2. This processing can be said to be processing that adds parallax corresponding to the viewing distance Vd rather than zeroing the parallax in the depth direction of the virtual display surface 11 in the shift processing described with reference to FIG. 13 , for example. In this way, the viewing distance Vd of the viewer 2 can be determined by signal processing rather than mechanically controlling the shooting viewpoint P (stereo camera 10).

[0140] 16A and 16B are schematic diagrams showing an example of a parallax adjustment process using image shifting. Here, as shown in Fig. 16A, the depth position Cd of the stereo camera 10 is set to 1 m at the shooting location. That is, the shooting viewpoint P of the stereo camera 10 is fixed at a position 1 m away from the virtual display surface 11. The height position Ch of the stereo camera 10 and the inter-camera distance ICD are adjustable.

[0141] The distance from the stereo camera 10 (shooting viewpoint P) to the subject, performer 1, is 1.5 m. Therefore, the stereoscopic image 5 generated from the images captured by the stereo camera 10 (right camera image 4R and left camera image 4L) is an image in which performer 1 is seen at a depth of 1.5 m. Note that, because the shooting viewpoint P cannot be moved in the depth direction, it is not possible to capture an image of performer 1 viewed from closer or farther away, for example.

[0142] 16B , it is assumed that the viewing distance Vd of viewer 2 at the viewing location is 1.3 m. That is, the viewpoint Q of viewer 2 is located 1.3 m away from the display surface 41 of the stereoscopic display device 40. Therefore, the distance between viewpoint Q and display surface 41 is 0.3 m longer than the distance between the shooting viewpoint P and virtual display surface 11.

[0143] Incidentally, when stereoscopic video 5 is displayed with the parallax adjusted to zero at the depth of virtual display surface 11, the stereoscopic video 5 can be viewed properly from a position 1 m away from display surface 41, and viewer 2 may feel uncomfortable at viewpoint Q where Vd = 1.3 m. Therefore, the parallax at the depth of virtual display surface 11 is adjusted according to Vd. This makes it possible to adjust the depth of the entire stereoscopic image 6 formed by stereoscopic video 5 forward or backward.

[0144] For example, as explained in the above embodiment, when the depth position Cd on the shooting side and the viewing distance Vd on the viewer side can be made the same, the parallax of the camera-side end of bowl 3 that overlaps with virtual display surface 11 is set to 0. In contrast, in the example shown in Fig. 16B, the parallax of the shift processing is adjusted so that the camera-side end of bowl 3 appears to be 0.3 m closer to the camera on virtual display surface 11.

[0145] 16C shows the depth of the stereoscopic image 6 when the stereoscopic video 5 is displayed with the parallax adjusted according to the viewing distance Vd. For example, the edge of the teacup 3 on the camera side appears to protrude 0.3 m from the display surface 41. Adjusting the parallax (depth) in this way makes it possible to display the performer 1 as if he or she were 1.5 m away from the viewer 2's viewpoint Q. This allows the stereoscopic video 5 to be viewed appropriately.

[0146] In this way, even if the depth position Cd on the shooting side and the viewing distance Vd on the viewer side are different, by appropriately adjusting the parallax of the stereoscopic image 5, it is possible to balance out the depth position and provide appropriate stereoscopic display. Conversely, because the depth position can be adjusted by image processing, it is possible to omit the depth direction adjustment mechanism. This allows for the size and cost of the shooting device to be reduced.

[0147] Note that when adjusting the depth position by shift processing, it may be difficult to accommodate depending on the amount of deviation between Cd and Vd. For example, when performing a pop-up display in a stereoscopic display, it is desirable to prevent the object to be displayed in the pop-up display from interfering with the frame of the display surface 41 of the stereoscopic display device 40. For example, if the object overlaps the frame of the display surface 41, the viewer 2 will perceive a deviation between the frame at the actual depth and the virtual depth of the object displayed in the pop-up display, which may cause 3D motion sickness or the like.

[0148] For example, if viewer 2 is further away from display surface 41 than shown in Fig. 16B, adjusting that distance by shifting may cause performer 1 to appear to pop out. In this case, the movement of performer 1 may interfere with the frame of display surface 41, potentially causing motion sickness or other discomfort. For this reason, it is preferable to adjust the depth position (parallax) by shifting within a range that does not interfere with the frame of display surface 41.

[0149] <Fourth Embodiment> Fig. 17 is a schematic diagram showing an example of the configuration of a 3D telepresence system according to a fourth embodiment. In a 3D telepresence system 400 according to this embodiment, viewing parameters are input via a predetermined UI (User Interface). Therefore, a UI information acquisition unit 34 is provided in the viewer controller 31. Note that the 3D telepresence system 400 does not include the viewer sensor 30 and viewer information measurement unit 33 described in the above embodiments.

[0150] The UI information acquisition unit 34 is a functional block configured by the viewer-side controller 31, and reads values ​​input into a predetermined UI for inputting viewing parameters (hereinafter referred to as input UI). As the input UI, for example, a GUI (Graphical User Interface) such as an input screen provided with input fields for the interpupillary distance IPD, viewing height Vh, viewing distance Vd, etc. of the viewer 2 is used. The input UI is displayed on, for example, a display connected to the viewer-side controller 31.

[0151] The input UI allows input of at least one of the interpupillary distance IPD, viewing height Vh, and viewing distance Vd. For example, the interpupillary distance IPD of the viewer 2 is measured in advance using an IPD measuring device or the like, and the measurement result is input via the input UI. In addition, the height of the viewer 2, the height of the viewpoint measured in a posture when viewing the stereoscopic video 5, or the like is input as the viewing height Vh. In addition, if the viewing position of the viewer 2 is fixed, the viewing distance Vd or the like according to the viewing position is input.

[0152] For example, before using the 3D telepresence system 400 or while waiting in line, information that serves as the viewing parameters of each viewer 2 is measured, and the measurement results are input from an input UI by a staff member or the like. Then, each time a viewer 2 changes, the corresponding viewing parameters are read, and adjustments are made to the shooting viewpoint P of the stereo camera 10. Even with this configuration, it is possible to provide an appropriate stereoscopic display that suits the state of the viewer 2.

[0153] Other Embodiments The present technology is not limited to the above-described embodiments, and various other embodiments can be realized.

[0154] In the above embodiment, the stereoscopic image 6 is displayed in actual size. However, the scale of the stereoscopic image 6 is not limited to actual size and can be set arbitrarily.

[0155] Fig. 18 is a schematic diagram showing an example of adjusting the inter-camera distance according to the scale of the stereoscopic image 6. In Fig. 18, the scale of the stereoscopic image 6 displayed by the stereoscopic display device 40 is set to X times the actual size, where X is a positive real number. For example, if 0 < X ​​< 1, the stereoscopic image 6 is displayed smaller than the actual size. Also, if 1 < X, the stereoscopic image 6 is displayed larger than the actual size. A case where the stereoscopic image 6 is displayed at such an X-times scale will be described.

[0156] In the example shown in Fig. 18, the inter-camera distance ICD is set to twice the interpupillary distance IPD of the viewer 2 (ICD = 2 x IPD). The stereoscopic image 6 displayed by the stereoscopic video 5 captured in this manner is at half the scale of actual size, as shown in the lower part of Fig. 18. This also applies to other scales. In other words, to display the stereoscopic image 6 at a scale X times larger than actual size, ICD = IPD / X is set.

[0157] Therefore, in this embodiment, the camera control unit 25 sets the inter-camera distance ICD of the stereo camera 10 to 1 / X times the interpupillary distance IPD of the viewer 2. This allows the 3D image 6 to be displayed at the scale intended by the imaging side. The scale of the 3D image 6 changes in all directions, including the X, Y, and Z directions. Therefore, for example, in FIG. 18 , the height, width, and distance from the viewer 2 of the 3D image 6 of the performer 1 are half that of when the 3D image 6 is displayed at life-size. Using this method, for example, a photograph of the performer 1 performing in front of a background drawn at double scale can be captured and displayed at half scale. In this case, only the size of the performer 1 can be displayed as if it were half. Additionally, the 3D image 6 can be reduced or enlarged to any desired size during display.

[0158] Fig. 19 is a schematic diagram showing an example of adjusting the height position of the stereo camera 10. In Fig. 19, the camera control unit 25 sets the height position Ch of the stereo camera 10 to a position different from the viewing height Vh of the viewer 2. In this case, the height of the stereo camera 10 can be freely set, making it possible to display the stereoscopic image 6 viewed from any height.

[0159] For example, the upper diagram of Fig. 19 is a schematic diagram of a stereoscopic image 6 that is seen when the height position Ch of the stereo camera 10 is set to the viewing height Vh of the viewer 2. In this case, the upper end of the flower 45, which is the subject, is outside the shooting range. In such a case, the height position Ch is adjusted independently of the viewing height Vh. For example, Ch is set to a higher value and the stereo camera 10 is moved upward. This makes it possible to fit the entire flower 45 within the shooting range, as shown in the lower diagram of Fig. 19. The height position Ch is adjusted by the viewer 2, for example, using a remote controller or the like.

[0160] In this way, by controlling the height position Ch of the stereo camera 10 to a height different from the viewing height Vh of the viewer 2, the viewer 2 can view the 3D image 6 at any height without having to move to change the viewing height Vh. For example, even if the viewer 2 remains seated, the viewer 2 can see the 3D image 6 at his / her feet or above him / her. This enables effective video presentation in use cases such as remote control.

[0161] 20 is a schematic diagram showing an example configuration of a communication device using a half mirror. The communication device 500 has a stereo camera 510, a stereoscopic display device 540, and a half mirror 550. The stereo camera 510 is arranged facing horizontally so as to capture an image of a user 501 who is also a viewer. The half mirror 550 is arranged between the stereo camera 510 and the user 501. The half mirror 550 is arranged at an angle (for example, 45°) with respect to the vertical plane, with its upper end tilted toward the stereo camera 510. The stereoscopic display device 540 is arranged above the half mirror 550 so as to be parallel to the horizontal direction.

[0162] For example, the stereoscopic video 5 displayed by the stereoscopic display device 540 is reflected by a half mirror 550 and viewed by the user 501. The stereo camera 510 can capture an image of the user 501 from the front via the half mirror 550. This allows the user 501 viewing the stereoscopic display device 540 to be captured by the stereo camera 510 from the front. By using such a communication device 500 at a remote location, communication by stereoscopic display with the users 501 aligning their gazes becomes possible.

[0163] In this way, the present technology can be applied to a system that aims for eye contact using the half mirror 550 by adding the optical characteristics of the half mirror 550 as a condition. For example, in Fig. 20, the intersection between the shooting direction of the stereo camera 510 and the half mirror 550 is denoted by M. Also, the intersection where a perpendicular line dropped from the intersection M to the stereoscopic display device 540 intersects with the display surface is denoted by D.

[0164] For example, if communication device 500 is considered to be a viewer device, the length of the path from viewpoint Q of user 501, passing through intersection M, to intersection D is the viewing distance Vd of user 501. The height of viewpoint Q of user 501 is the viewing height Vh. Note that the interpupillary distance IPD of user 501 can be measured in the same way as in the above embodiment. This makes it possible to measure viewing parameters.

[0165] Furthermore, if the communication device 500 is considered to be the device on the imaging side, the length of the path from the imaging viewpoint P of the stereo camera 510, passing through the intersection point M, to the intersection point D is the depth position Cd of the stereo camera 510. The height of the imaging viewpoint P of the stereo camera 510 is the height position Ch. Note that the inter-camera distance ICD is the distance between the left and right cameras, as in the above embodiment.

[0166] These adjustment parameters (the inter-camera distance ICD, the height position Ch, and the depth position Cd) are mechanically adjusted based on, for example, the viewing parameters of the communication partner, thereby enabling, for example, eye contact while realizing communication through a life-size stereoscopic display with a sense of ground contact.

[0167] Furthermore, although the above description has mainly focused on the case where a stereoscopic display device equipped with a display panel or the like is used, the present invention is not limited to this, and a device that performs stereoscopic display using a projector or the like may also be used. Even when a projector is used, the shooting viewpoint P of the stereo camera is mechanically adjusted according to the positional relationship between the viewer and the IPD or the display surface (projection surface) of the viewer. In addition, the present technology can be applied to any stereoscopic display system in which the display is adjusted according to the characteristics of the viewer and the positional relationship between the viewer and the display surface.

[0168] In the above, the information processing method according to the present technology is executed by a control device that is a processor of an imaging-side controller provided at a base of the imaging side. However, without being limited to this, the information processing method and program according to the present technology may be executed by linking the imaging-side controller with another computer that can communicate via a network or the like, thereby constructing an information processing device according to the present technology.

[0169] In other words, the information processing method and program according to the present technology can be executed not only in a computer system composed of a single computer, but also in a computer system in which multiple computers operate in conjunction with each other. In this disclosure, a system refers to a collection of multiple components (devices, modules (parts), etc.), regardless of whether all the components are in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device housed in a single housing with multiple modules, are both systems.

[0170] The execution of the information processing method and program according to the present technology by a computer system includes both cases where control for adjusting the shooting viewpoint of a stereo camera is executed by a single computer and cases where each process is executed by a different computer. Furthermore, the execution of each process by a specific computer includes having another computer execute part or all of the process and obtaining the results.

[0171] That is, the information processing method and program according to the present technology can also be applied to a cloud computing configuration in which a single function is shared and processed jointly by multiple devices via a network.

[0172] It is also possible to combine at least two of the features of the present technology described above. That is, the various features described in each embodiment may be arbitrarily combined without distinguishing between the embodiments. Furthermore, the various effects described above are merely examples and are not intended to be limiting, and other effects may also be achieved.

[0173] In the present disclosure, the terms "same," "equal," "orthogonal," etc. are concepts that include "substantially the same," "substantially equal," "substantially orthogonal," etc. For example, they also include states that fall within a predetermined range (e.g., a range of ±10%) based on "completely the same," "completely equal," "completely orthogonal," etc.

[0174] The present technology can also be configured as follows. (1) An information processing device including a camera control unit that mechanically controls stereo cameras to adjust the shooting viewpoints of the stereo cameras based on viewing parameters including at least one of an interpupillary distance of a viewer viewing a stereoscopic display device that displays stereoscopic images captured using stereo cameras in real time, a viewing height of the viewer, and a viewing distance of the viewer. (2) The information processing device described in (1), wherein the stereo cameras are connected to an adjustment mechanism that mechanically changes at least one of an inter-camera distance of the stereo cameras, a height position of the stereo cameras, and a depth position of the stereo cameras, and the camera control unit controls the adjustment mechanism based on the viewing parameters. (3) The information processing device described in (2), wherein the viewing parameters include an inter-pupillary distance of the viewer, the adjustment mechanism is capable of adjusting the inter-camera distance of the stereo cameras, and the camera control unit controls the adjustment mechanism to adjust the inter-camera distance based on the inter-pupillary distance of the viewer. (4) The information processing device according to (2) or (3), wherein the viewing parameters include a viewing height of the viewer, the adjustment mechanism is capable of adjusting a height position of the stereo cameras, and the camera control unit controls the adjustment mechanism to adjust the height position based on the viewing height of the viewer. (5) The information processing device according to at least one of (2) to (4), wherein the viewing parameters include a viewing distance of the viewer, the adjustment mechanism is capable of adjusting a depth position of the stereo cameras, and the camera control unit controls the adjustment mechanism to adjust the depth position based on the viewing distance of the viewer. (6) The information processing device according to at least one of (2) to (5), further comprising an image generation unit that generates in real time the stereoscopic images to be displayed on the stereoscopic display device from images captured by the stereo cameras with the adjusted shooting viewpoints.(7) The information processing device according to (6), wherein the image generation unit performs a shift process to adjust parallax on the images captured by the stereo cameras, a trimming process to fit the angle of view to a display range of the stereoscopic display device, and an enlargement process to fit the images to a predetermined image size, thereby generating the stereoscopic images. (8) The information processing device according to (6) or (7), wherein a scale of the stereoscopic image displayed by the stereoscopic display device is set to actual size, and the camera control unit controls the adjustment mechanism so that a positional relationship between a virtual display surface, which is an image surface of the stereoscopic display device in a shooting space of the stereo cameras, and the shooting viewpoint of the stereo cameras becomes the same as a positional relationship between the display surface of the stereoscopic display device and the viewpoint of the viewer. (9) The information processing device according to (8), wherein the viewing parameters include the interpupillary distance of the viewer, the viewing height of the viewer, and the viewing distance of the viewer, the stereo cameras are adjustable in their inter-camera distance, height position, and depth position, and the camera control unit: matches the inter-camera distance of the stereo cameras to the interpupillary distance of the viewer, matches the height position of the stereo cameras to the viewing height of the viewer, and matches the distance between the stereo cameras and the virtual display surface to the viewing distance of the viewer. (10) The information processing device according to (8) or (9), wherein the image generation unit shifts the images captured by the stereo cameras so that parallax in the depth of the virtual display surface becomes zero. (11) The information processing device according to (10), wherein the image generation unit crops an area corresponding to the virtual display surface as seen from the adjusted shooting viewpoint from the shifted image, and enlarges the cropped image to a predetermined image size. (12) The information processing device according to at least one of (6) to (11), wherein the adjustment mechanism has a position sensor that detects the position of the shooting viewpoint of the stereo camera, and the image generation unit adjusts the image captured by the stereo camera based on a difference between an adjustment value of the shooting viewpoint according to the viewing parameters and a detection value of the position sensor.(13) The information processing device according to at least one of (6) to (12), wherein the viewing parameters include a viewing distance of the viewer, and the image generation unit shifts the images captured by the stereo cameras so that a parallax in the depth of the virtual display surface corresponds to the viewing distance. (14) The information processing device according to at least one of (1) to (13), wherein a scale of a stereoscopic image displayed by the stereoscopic display device is X times the actual size, where X is a positive real number, the viewing parameters include the interpupillary distance of the viewer, and the camera control unit sets the inter-camera distance of the stereo cameras to 1 / X times the interpupillary distance of the viewer. (15) The information processing device according to at least one of (1) to (14), wherein the viewing parameters include a viewing height of the viewer, and the camera control unit sets the height position of the stereo cameras to a position different from the viewing height of the viewer. (16) The information processing device according to at least one of (1) to (15), wherein the viewing parameters are detected by a sensor that measures the viewer. (17) The information processing device according to at least one of (1) to (16), wherein the viewing parameters are input via a predetermined UI. (18) An information processing method executed by a computer system, wherein the stereo cameras are mechanically controlled to adjust the shooting viewpoint of the stereo cameras based on viewing parameters including at least one of an interpupillary distance of a viewer viewing a stereoscopic display device that displays stereoscopic images captured using the stereo cameras in real time, a viewing height of the viewer, and a viewing distance of the viewer. (19) A computer-readable recording medium having recorded thereon a program for causing a computer to execute the step of mechanically controlling the stereo cameras to adjust the shooting viewpoint of the stereo cameras based on viewing parameters including at least one of an interpupillary distance of a viewer viewing a stereoscopic display device that displays stereoscopic images captured using the stereo cameras in real time, a viewing height of the viewer, and a viewing distance of the viewer.

[0175] P: Shooting viewpoint 2: Viewer 5: Stereoscopic image 10: Stereo camera 11: Virtual display surface 12: Adjustment mechanism 24: Control device 25: Camera control unit 26: Image processing unit 30: Viewer sensor 40: Stereoscopic display device 41: Display surface 100, 200, 300, 400: 3D telepresence system

Claims

1. An information processing device comprising: a camera control unit that mechanically controls a stereo camera to adjust the shooting viewpoint of the stereo camera based on viewing parameters including at least one of the interpupillary distance of a viewer viewing a stereoscopic display device that displays stereoscopic images captured using the stereo camera in real time, the viewing height of the viewer, and the viewing distance of the viewer.

2. An information processing device according to claim 1, wherein the stereo camera is connected to an adjustment mechanism that mechanically changes at least one of the inter-camera distance, the height position, and the depth position of the stereo camera, and the camera control unit controls the adjustment mechanism based on the viewing parameters.

3. An information processing device according to claim 2, wherein the viewing parameters include the interpupillary distance of the viewer, the adjustment mechanism is capable of adjusting the inter-camera distance of the stereo camera, and the camera control unit controls the adjustment mechanism to adjust the inter-camera distance based on the interpupillary distance of the viewer.

4. An information processing device as described in claim 2, wherein the viewing parameters include a viewing height of the viewer, the adjustment mechanism is capable of adjusting a height position of the stereo camera, and the camera control unit controls the adjustment mechanism to adjust the height position based on the viewing height of the viewer.

5. An information processing device according to claim 2, wherein the viewing parameters include a viewing distance of the viewer, the adjustment mechanism is capable of adjusting a depth position of the stereo camera, and the camera control unit controls the adjustment mechanism to adjust the depth position based on the viewing distance of the viewer.

6. An information processing device according to claim 2, further comprising an image generating unit that generates in real time the stereoscopic image to be displayed on the stereoscopic display device from the images captured by the stereo camera with the shooting viewpoint adjusted.

7. An information processing device according to claim 6, wherein the image generating unit generates the stereoscopic image by performing a shift process for adjusting parallax on the images captured by the stereo camera, a trimming process for matching the angle of view to the display range of the stereoscopic display device, and an enlargement process for matching the images to a predetermined image size.

8. An information processing device according to claim 6, wherein the scale of the stereoscopic image displayed by the stereoscopic display device is set to actual size, and the camera control unit controls the adjustment mechanism so that the positional relationship between a virtual display surface that embodies the display surface of the stereoscopic display device in the shooting space of the stereo camera and the shooting viewpoint of the stereo camera becomes the same as the positional relationship between the display surface of the stereoscopic display device and the viewpoint of the viewer.

9. An information processing device according to claim 8, wherein the viewing parameters include the interpupillary distance of the viewer, the viewing height of the viewer, and the viewing distance of the viewer; the stereo camera is capable of adjusting an inter-camera distance, a height position, and a depth position; and the camera control unit: matches the inter-camera distance of the stereo camera to the interpupillary distance of the viewer; matches the height position of the stereo camera to the viewing height of the viewer; and matches the distance between the stereo camera and the virtual display surface to the viewing distance of the viewer.

10. An information processing device according to claim 8, wherein the image generating section shifts the images captured by the stereo cameras so that parallax in the depth direction of the virtual display surface becomes zero.

11. An information processing device according to claim 10, wherein the image generating unit crops an area from the shifted image that corresponds to the virtual display surface as seen from the adjusted shooting viewpoint, and enlarges the cropped image to a predetermined image size.

12. An information processing device according to claim 6, wherein the adjustment mechanism has a position sensor that detects the position of the shooting viewpoint of the stereo camera, and the image generation unit adjusts the image captured by the stereo camera based on the difference between an adjustment value of the shooting viewpoint corresponding to the viewing parameters and the detection value of the position sensor.

13. An information processing device according to claim 6, wherein the viewing parameters include a viewing distance of the viewer, and the image generating unit shifts the images captured by the stereo camera so that the parallax in the depth direction of the virtual display surface corresponds to the viewing distance.

14. An information processing device according to claim 1, wherein a scale of a stereoscopic image displayed by the stereoscopic display device is X times the actual size, where X is a positive real number, the viewing parameters include the interpupillary distance of the viewer, and the camera control unit sets the inter-camera distance of the stereo cameras to 1 / X times the interpupillary distance of the viewer.

15. An information processing device according to claim 1, wherein the viewing parameters include a viewing height of the viewer, and the camera control unit sets a height position of the stereo camera to a position different from the viewing height of the viewer.

16. An information processing device according to claim 1, wherein the viewing parameters are detected by a sensor that measures the viewer.

17. An information processing device according to claim 1, wherein the viewing parameters are input via a predetermined UI.

18. An information processing method executed by a computer system, which mechanically controls a stereo camera to adjust the shooting viewpoint of the stereo camera based on viewing parameters including at least one of the interpupillary distance of a viewer viewing a stereoscopic display device that displays stereoscopic images taken using the stereo camera in real time, the viewing height of the viewer, and the viewing distance of the viewer.

19. A computer-readable recording medium having recorded thereon a program for executing a step of mechanically controlling the stereo camera so as to adjust the shooting viewpoint of the stereo camera based on viewing parameters including at least one of the interpupillary distance of a viewer viewing a stereoscopic display device that displays stereoscopic images taken using the stereo camera in real time, the viewing height of the viewer, and the viewing distance of the viewer.

Images