Display device for virtual space

Abstract

To provide an excellent virtual reality experience at a low cost. [Solution] The virtual space display device 1 according to the present invention comprises a face position identification unit 113 that identifies the position of the viewer's face, a viewpoint identification unit 114 that identifies the position of the left eye viewpoint and the position of the right eye viewpoint, an image generation unit 115 that generates a left-eye image of the virtual space as seen from the left-eye viewpoint and a right-eye image of the virtual space as seen from the right-eye viewpoint, and an image display unit 116 that displays the virtual space in 3D on a display screen based on the left-eye image and the right-eye image. The image display unit 116 displays the above-mentioned virtual space in 3D in one or more modes specified by the user from the group which includes at least a mode of viewing in 3D through active shutter glasses, a mode of viewing in 3D through anaglyph glasses, and a mode of viewing in 3D with polarized glasses or the naked eye.

Description

Technical Field

[0001] The present invention relates to a display device for a virtual space.

Background Art

[0002] There is known a technique for generating an image representing a three-dimensional object observed from a specified observation position, exemplified by three-dimensional computer graphics processing technology and the like. Further, there is known a technique for detecting the position of an observer who observes a display screen on which an image representing a three-dimensional object is displayed, and generating and displaying on the display screen an image representing the three-dimensional object that should be observed from the detected position.

[0003] In conventional services and prior arts, regarding a technique for stereoscopically displaying a virtual space, Patent Document 1 discloses an image generation device that outputs an image representing a three-dimensional object to an external display device, the image generation device including: detection means for detecting an observation position of an observer who observes an image displayed by the display device; position calculation means for calculating a virtual viewpoint obtained by multiplying a displacement amount from a predetermined reference position facing a display area of the image displayed by the display device to the observation position detected by the detection means by r (r is a real number greater than 1); generation means for acquiring data for generating an image representing the three-dimensional object and generating an image representing the three-dimensional object observed from the virtual viewpoint calculated by the position calculation means; and output means for outputting the image generated by the generation means to the display device.

[0004] The technology described in Patent Document 1 detects the observation position based on the positions of the right and left eyes in real space, which are calculated by the eye position calculation function of the head tracking unit. Patent Document 1 also mentions a 3D television that uses active shutter or polarizing glasses as a display device for stereoscopic display. Regarding the generation of images representing three-dimensional objects observed from a virtual viewpoint, Patent Document 1 shows a procedure in which each of the objects that virtually exist in the virtual space is unfolded in the virtual space, and shadows are applied to each of the unfolded objects using a set light source.

[0005] The technology described in Patent Document 1 makes it possible to generate an image such that the amount of movement required by the observer is less than in conventional methods when attempting to change the observation angle of an object represented in the image. [Prior art documents] [Patent Documents]

[0006] [Patent Document 1] International Publication No. 2012 / 147363 [Overview of the Initiative] [Problems that the invention aims to solve]

[0007] Incidentally, there is a demand for relatively low-cost 3D display of virtual spaces. The technology described in Patent Document 1 is limited to 3D televisions that use active shutter or polarized glasses as display devices, so users who do not own such equipment will incur additional costs to implement them.

[0008] Therefore, the technology described in Patent Document 1 has room for further improvement in terms of providing a superior virtual space experience at a low cost.

[0009] The objective of this invention is to provide an excellent virtual space experience at a low cost. [Means for solving the problem]

[0010] As a result of diligent research to solve the above problems, the inventors of the present invention found that the above objectives could be achieved by allowing users to select the mode of three-dimensional display according to the equipment they possess. Thus, the inventors of the present invention were completed.

[0011] One aspect of the present invention provides a virtual space display device comprising: a face position identification unit that identifies the position of the viewer's face; a viewpoint identification unit that identifies the position of the left eye viewpoint and the position of the right eye viewpoint relating to a virtual space to be displayed on a display screen based on the face position; an image generation unit that generates a left-eye image of the virtual space as seen from the left-eye viewpoint and a right-eye image of the virtual space as seen from the right-eye viewpoint; and an image display unit that displays the virtual space in 3D on the display screen based on the left-eye image and the right-eye image, wherein the image display unit displays the virtual space in 3D in one or more modes specified by the user from the group including at least a mode of viewing in 3D through active shutter glasses, a mode of viewing in 3D through anaglyph glasses, and a mode of viewing in 3D with polarized glasses or the naked eye.

[0012] While the technology described in Patent Document 1 limits the display device to a 3D television that uses relatively expensive glasses, this embodiment allows for the display of a virtual space in 3D in one or more ways specified by the user from the group which includes at least one way of viewing in 3D through active shutter glasses, one way of viewing in 3D through anaglyph glasses, and one way of viewing in 3D through polarized glasses or with the naked eye.

[0013] This means that the embodiment can provide 3D display of virtual space by utilizing various existing equipment owned by the user. Furthermore, this embodiment can provide 3D display of virtual space using a general display already owned by the viewer, with only the need for the viewer to procure relatively inexpensive equipment such as anaglyph glasses.

[0014] Based on the above, this embodiment can provide an excellent virtual space experience at a low cost.

[0015] Another aspect of the present invention is an aspect relating to the above-described aspect, wherein the image display unit displays the virtual space in three dimensions on a display screen projected by a projector, the display device further comprises an operation acquisition unit that acquires touch operations on the display screen from an external device different from the projector, and the viewpoint identification unit, in the case of a one-finger touch operation, horizontally moves the left-eye viewpoint and the right-eye viewpoint in accordance with the movement of the finger, in the case of a two-finger touch operation, changes the direction of the line segment with the left-eye viewpoint and the right-eye viewpoint as its ends and the scale of the virtual space in accordance with changes in the direction and length of the line segment with the two fingers as its ends, and in the case of a three-finger touch operation, vertically moves the left-eye viewpoint and the right-eye viewpoint in accordance with the movement of the three fingers, providing a display device.

[0016] Touch controls that allow users to freely change their viewpoint, along with a large display screen, enhance the virtual space experience. However, large flat-screen displays equipped with touch panels are expensive to implement. While there are methods to achieve this using head-mounted displays, implementation costs remain a challenge even with this approach.

[0017] This embodiment enables 3D display on a large screen at a relatively low cost by using a projector for stereoscopic display. Furthermore, this embodiment enables operation that allows the viewpoint to be freely changed by touching the display screen, even when using an inexpensive projector that does not provide touch operation, by acquiring touch operation from an external device different from the projector.

[0018] In addition, this embodiment provides an intuitive operating system that switches between horizontal movement, rotation, and scaling of the viewpoint, as well as vertical movement of the viewpoint, using the number of fingers used in touch operations. This reduces the cost of introducing a separate input device to switch between types of operations while providing an excellent virtual space experience.

[0019] Based on the above, this embodiment can provide an excellent virtual space experience at a low cost.

[0020] In addition, the present invention can take various aspects such as changing the speed of viewpoint movement according to the scale to provide a more intuitive viewpoint movement, displaying the terrain on a substantially horizontal display screen to provide an excellent virtual space experience like strolling through a diorama, pasting a real-world background on a hollow sphere to provide an excellent virtual space experience using real images, specifying the viewpoint based on data from an external device to reduce the computational cost related to viewpoint specification, using multi-displays arranged around the viewer to provide an immersive stereoscopic display, and performing stereoscopic display in anaglyph to provide a virtual space experience with only low-cost red-blue glasses. These aspects contribute to providing an excellent virtual space experience at low cost by the effects brought about by adding their respective unique configurations.

Advantages of the Invention

[0021] As described above, the present invention can provide an excellent virtual space experience at low cost.

Brief Description of the Drawings

[0022] [Figure 1] FIG. 1 is a diagram showing an overview of the system S of the present embodiment. [Figure 2] FIG. 2 is a block diagram showing an example of the hardware configuration and software configuration of the system S of the present embodiment. [Figure 3] FIG. 3 is an example of the three-dimensional model database 131. [Figure 4] FIG. 4 is a main flowchart showing an example of a preferred flow of display processing executed by the display device 1 of the present embodiment. [Figure 5] FIG. 5 is a figure following the previous figure. [Figure 6] FIG. 6 is a figure following the previous figure. [Figure 7] FIG. 7 is a figure following the previous figure. [Figure 8] FIG. 8 is a figure following the previous figure.

Embodiments for Carrying Out the Invention

[0023] Firstly, although the following disclosures, figures, and / or claims are described either individually or in combination with one or more other aspects, the subject matter of the immediate disclosure is not intended to be limited in that way. That is, the immediate disclosures, figures, and claims are intended to encompass the various aspects described herein, either individually or in one or more combinations with each other. For example, even if the immediate disclosure describes and illustrates the first, second, and third embodiments in such a way that the first embodiment is described and illustrated particularly in relation to the second embodiment, or the second embodiment is described and illustrated only in relation to the third embodiment, the immediate disclosures and illustrations are not limited in that way and may include only the first embodiment, only the second embodiment, only the third embodiment, or one or more combinations of the first, second, and / or third embodiments, such as the first and second embodiments, the first and third embodiments, the second and third embodiments, or the first, second, and third embodiments.

[0024] In this text, the phrase "or" is used to mean a "non-exclusive" arrangement unless explicitly specified otherwise. For example, when we say "item x is A or B," it means either (1) item x is either A or B, or (2) item x is both A and B. In other words, the word "or" is not used to define an "exclusive" arrangement.

[0025] Furthermore, when the phrases "contain at least one" or "contain at least one of the following" are used in the text, they mean that the system or element contains one or more of the elements listed after the phrase. For example, if there are three types of elements, from element 1 to element 3, the phrases "contain at least one" or "contain at least one of the following" are interpreted as any of the following structural arrangements: a device containing element 1, a device containing element 2, a device containing element 3, a device containing element 1 and element 2, a device containing element 1 and element 3, a device containing element 2 and element 3, or a device containing element 1, element 2, and element 3.

[0026] The same interpretation is intended when the phrase "used in at least one of the following" is used in the text. Furthermore, "and / or" as used in the text is used as a linguistic conjunction to indicate that one or more of the listed elements or conditions are included or occur. For example, a device containing the first element, the second element, and / or the third element is interpreted as any of the following structural arrangements: a device containing the first element, a device containing the second element, a device containing the third element, a device containing the first and second elements, a device containing the first and third elements, a device containing the second and third elements, or a device containing the first, second, and third elements.

[0027] Furthermore, the use of the phrase "and / or" in this text signifies a "non-exclusive" arrangement, as stipulated in the Japanese Industrial Standard (JIS) "Format and Preparation Method of Standards Documents JIS Z 8301".

[0028] The following describes in detail an example of an embodiment of the present invention with reference to the drawings.

[0029] <System S> Figure 1 is a schematic diagram of the system S of this embodiment. The virtual space display system (system S) according to this embodiment consists of a virtual space display device 1, a camera C, a display D, and stereoscopic glasses G. The display device 1 is configured to communicate with the camera C and the display D via a network N. In the example in Figure 1, the camera C, which is a commercially available webcam, is mounted on the top of the display D.

[0030] In system S, the display D displays an image created by combining a foreground image F and a background image B, which can be viewed in 3D by viewer A through stereoscopic glasses G, through display processing performed in the display device 1. In the example in Figure 1, the foreground image F, which depicts a long-haired woman with her left hand extended towards viewer A, is superimposed on the background image B, which depicts a school corridor. In this example, the parts of the foreground image F and background image B that are not displayed on the display D are indicated by dotted lines. As shown in this example, the stereoscopic glasses G may have a handle H for viewer A to grasp and an infrared emitter E that emits infrared rays.

[0031] Figure 2 is a block diagram showing an example of the hardware and software configuration of system S in this embodiment. The following describes an example of the hardware and software configuration of system S in this embodiment using Figure 2 and other figures.

[0032] [Display device 1] The display device 1 comprises various hardware components such as a control unit 11, a storage unit 13, and a communication unit 14. The display device 1 performs a series of processes to display the virtual space in 3D by combining a background image B rendered inside a hollow sphere surrounding the virtual space with a foreground image F rendered from a 3D model placed inside the hollow sphere.

[0033] The form of the display device 1 is not particularly limited. A preferred example of a device as the display device 1 is a relatively inexpensive home computer, such as a personal computer, notebook computer, or tablet terminal. Designing and manufacturing the display device 1 as a dedicated device specialized for the display processing described later is not preferable, as it would contradict the objective of displaying a virtual space in 3D using low-cost equipment.

[0034] [Control Unit 11] The control unit 11 includes a Central Processing Unit (CPU), Random Access Memory (RAM), and Read Only Memory (ROM), among other things.

[0035] The control unit 11 cooperates with at least one of the storage unit 13 and the communication unit 14 as needed. The control unit 11 then implements the software components of the program of this embodiment executed on the display device 1, such as the image acquisition unit 111, the light-emitting element position acquisition unit 112, the face position identification unit 113, the viewpoint identification unit 114, the video generation unit 115, the video display unit 116, the operation acquisition unit 117, etc.

[0036] The details of the display processing implemented by the software components described above will be explained later using Figures 4 to 8.

[0037] [Storage section 13] The storage unit 13 is a device on which data and / or files are stored, and has a storage unit that stores data non-temporarily using a hard disk, semiconductor memory, recording medium, and memory card, etc. The storage unit 13 stores programs executed by a microcomputer, a 3D model database 131, etc.

[0038] The details of the display process realized by executing the above-mentioned program on the display device 1 will be explained later using Figures 4 to 8.

[0039] (3D Model Database 131) The 3D model database 131 stores data of the 3D model rendered by the display device 1 of this embodiment (3D model data). Although not essential, it is preferable that the 3D model data is stored in association with information that identifies the data (e.g., a model ID). If some or all of the 3D model data is managed on a file system, it is preferable that the 3D model data includes data relating to the resource name.

[0040] The 3D model data preferably includes data relating to the type, shape, and texture of the 3D model. The type may include data indicating whether it is a foreground or background. The shape may be a conventionally known shape, exemplified by, for example, the shape of a basic solid such as a hollow sphere, a shape formed by combining polygons, or a shape represented by an implicit function surface (e.g., a metaball). The texture may be a conventionally known texture, such as a texture relating to color or pattern, a texture relating to transparency, a texture relating to reflection, or a texture relating to surface shape.

[0041] If the 3D model is a movable model, it is preferable that the 3D model data include data related to the movement of the movable model. Examples of such data include data related to bones and data related to weights. Here, "weight" refers to a parameter that sets the effect that the movement of each bone has on the 3D shape.

[0042] To provide an immersive 3D display of a virtual space, it is preferable that the 3D model database 131 includes data related to the background based on images or videos of actual scenery. This allows the display device 1 to provide an immersive 3D display by combining a foreground image F with a background image B that closely resembles the actual scenery.

[0043] Figure 3 shows an example of the 3D model database 131. This example stores model data identified by model IDs "M0001," "M0002," and "M0003." While this example stores so-called character data, the 3D model database 131 of this embodiment is not limited to storing character data, but can store various 3D model data corresponding to the virtual space experience provided.

[0044] The first model data identified by model ID "M0001" is a 3D model data for a background that projects a school corridor scene onto the inside of a hollow sphere. It has the model type "background" and the resource name "background.obj," and includes data such as the shape "hollow sphere" (inside only, no outside rendering) and the texture "school corridor scenery." This "school corridor scenery" was generated based on live-action footage of a school corridor.

[0045] The second model data identified by model ID "M0002" is data for a foreground human model configured to be movable via bones, including data such as model type "Foreground (Movable Model)" and resource name "Woman.obj", as well as data such as shape, bones, and textures related to the 3D model of the woman in foreground image F in the example in Figure 1.

[0046] The third model data identified by model ID "M0003" is data for a large prop model for the foreground, including data such as model type "Foreground (Floor)" and resource name "Floor.obj", as well as data such as shape, bones, and texture related to the 3D model of the floor from which the shadow of the woman in foreground image F falls in the example in Figure 1.

[0047] Because these model data are stored in the 3D model database 131 of this example, the display device 1 can generate a right-eye image and a left-eye image by combining a background image B drawn using the first model data and a foreground image F of the floor depicting a woman drawn using the second model data and the shadow of the woman drawn using the third model data.

[0048] [Communications Section 14] The communication unit 14 is not particularly limited as long as it connects the display device 1 to the network N and enables communication. Examples of the communication unit 14 include a network card compatible with the Ethernet standard, a communication device compatible with wireless LAN, and a communication device compliant with the Bluetooth® standard.

[0049] [Camera C] Camera C is configured to include an optical sensor that provides data for identifying the face position of viewer A. It has an optical sensor that detects the infrared emitter E (described later) or an optical sensor that captures viewer A's face. The optical sensor that detects the infrared emitter E may be, for example, an infrared sensor or an infrared camera. The optical sensor that captures viewer A's face may be, for example, a commercially available webcam.

[0050] If the stereoscopic glasses G described later have an infrared emitter E, it is preferable that the camera C is configured to include an optical sensor for detecting the infrared emitter E. This allows the display device 1 to determine the face position by a process that detects the position of the infrared emitter E based on data from an infrared sensor, which can be implemented with less computation than image recognition processing that detects the face position from image data captured by a webcam or the like.

[0051] Instead of camera C, various external devices that provide data to determine the face position of viewer A may be used. Examples of such devices include VIVE Tracker®, KINECT®, and motion capture cameras. VIVE Tracker determines viewer A's position by using the timing when an infrared laser emitted by an external device (base station) installed in a fixed position (e.g., a corner of the room) hits a light-receiving sensor worn by viewer A.

[0052] [Display D] Display D is not particularly limited. For example, Display D may be a liquid crystal display, an organic electroluminescent display (OLED display), a projector, etc. In the following description, Display D will also be referred to as the display screen.

[0053] [Using a projector] To enable 3D display of a virtual space on a large screen using lower-cost equipment, it is preferable that display D includes a projector. This enables low-cost 3D display on a large screen.

[0054] [Using the screen of display device 1] When the display device 1 is configured with a device having a screen, such as a laptop computer or tablet terminal, it is preferable that the display D includes such a screen in order to display the virtual space in 3D with even lower-cost equipment.

[0055] [Immersive 3D display using multiple displays] In order to display the virtual space not only in front of viewer A but also around them, such as to the left, right, or behind, it is preferable that there be multiple displays D. In other words, it is preferable that the displays D be configured as a multi-display. In this case, in order to display the virtual space around viewer A, it is preferable that the multiple displays D be arranged to surround viewer A.

[0056] (Low-cost immersive 3D display using multiple screens) To achieve immersive 3D display on a large screen while further reducing equipment costs, it is preferable that the multiple displays D be a multi-screen setup using projectors and screens. Projectors and screens can achieve the same display size at a lower cost than using large display devices. Therefore, a multi-screen setup can achieve immersive 3D display at a lower cost.

[0057] [Stereoscopic Glasses G] Stereoscopic glasses G are glasses that work in cooperation with the display device 1 to realize stereoscopic viewing of stereoscopic images displayed on the display D. Examples of stereoscopic glasses G include active shutter glasses and anaglyph glasses.

[0058] [Handle H] It is preferable that the stereoscopic glasses G have a handle H for the viewer A to grasp. This allows viewer A to use the stereoscopic glasses G without their skin or other body parts coming into contact with it. Therefore, stereoscopic glasses G with a handle H make the viewing experience even more hygienic in stereoscopic display demonstrations aimed at an unspecified number of people.

[0059] [Infrared emitter E] When camera C detects infrared light, it is preferable that the stereoscopic glasses G have an infrared emitter E. This allows the display device 1 to determine the position of viewer A's face with relatively low processing load, by detecting the position of the infrared emitter E using camera C which has an infrared sensor or the like.

[0060] It is preferable that the infrared emitter E is provided together with the handle H. When the infrared emitter E is provided on stereoscopic glasses G, its weight may cause the temples or nosepieces of the glasses to press against the ears or nose of the viewer A. By providing the infrared emitter E together with the handle H, such discomfort caused by pressure can be reduced. Figure 1 shows an example in which the infrared emitter E is provided near the upper end of the handle H. By providing the infrared emitter E in this manner, the infrared rays it emits will not be blocked by the viewer A's hands.

[0061] [QR code] It is preferable that the stereoscopic glasses G are provided with a two-dimensional code for accessing a page to download the program of this embodiment. This allows viewer A to take a picture of the two-dimensional code on the stereoscopic glasses G with a camera-equipped terminal, such as a smartphone, and have the terminal execute the display processing as a display device 1.

[0062] [Active shutter glasses] To achieve high-color-reproduction stereoscopic display with equipment that is less expensive than displays that support naked-eye stereoscopic viewing, the stereoscopic glasses G are preferably active-shutter type glasses. Active-shutter type glasses are glasses that control shutters provided on lenses corresponding to each eye so that the two eyes open and close alternately.

[0063] In stereoscopic display using active shutter glasses, the right-eye image is displayed on display D when the right-eye shutter opens, and the left-eye image is displayed on display D when the left-eye shutter opens, thereby achieving stereoscopic display. The shutters are configured, for example, as liquid crystal shutters. The synchronization of the shutter opening and closing timing with the display is achieved by appropriate technical means.

[0064] [Anaglyph Glasses] To enable 3D display of virtual space using equipment that is even less expensive than active shutter glasses, the stereoscopic glasses G are preferably anaglyph glasses. Anaglyph glasses are glasses with lenses of different colors worn on each eye. When the lens corresponding to one eye is red-based and the lens corresponding to the other eye is blue-based, the anaglyph glasses are also called red-blue glasses.

[0065] Anaglyph glasses, like active shutter glasses, can achieve stereoscopic display of virtual spaces without requiring a display that supports naked-eye stereoscopic viewing. In stereoscopic display using anaglyph glasses, stereoscopic display is achieved by displaying an anaglyph image on display D, which is a composite of a right-eye image with a color that easily passes through the right eye lens and a left-eye image with a color that easily passes through the left eye lens.

[0066] While anaglyph glasses have disadvantages compared to active shutter glasses, such as inferior color reproduction and a tendency to produce crosstalk or ghost images, they are superior because they can be constructed relatively inexpensively without requiring electronic circuits, liquid crystal shutters, or power supplies.

[0067] [Operation acquisition device P] The system S preferably includes an operation acquisition device P that detects touch operations on the display screen and provides them to the display device 1. Examples of the operation acquisition device P include a device that detects touch operations using an optical sensor such as a touch operation detection sensor using laser light, and a device that detects touch operations using an acoustic sensor such as a touch operation detection sensor using ultrasound.

[0068] Touch controls that allow users to freely change their viewpoint, along with large display screens, enhance the virtual space experience. However, large flat-screen displays and head-mounted displays equipped with touch panels have high implementation costs.

[0069] Because the system S is configured to include an operation acquisition device P, the display device 1 can detect touch operations on a relatively inexpensive display D that does not provide touch operation. In particular, when the display D is implemented by a projector, the display device 1 acquires touch operations from an operation acquisition device P that is different from the projector, which normally does not provide touch operation. This enables the user to freely change their viewpoint by touching the display screen, even when using an inexpensive projector.

[0070] [Network N] The type of network N is not particularly limited as long as it enables communication for the display device 1, etc. Examples of network N include the internet, mobile phone networks, wireless LANs, and communication networks related to the Bluetooth® standard.

[0071] [Network N, specialized for connecting Display D] Network N may also include cables or devices that connect display device 1 and display D, as exemplified by cables compliant with the High-Definition Multimedia Interface (HDMI®) standard.

[0072] [Network N, specialized for connecting camera C] Network N may include cables or devices that connect the display device 1 and camera C, as exemplified by cables conforming to the Universal Serial Bus (USB) standard.

[0073] [Main flowchart of display processing] Figure 4 is a main flowchart showing an example of a preferred flow of display processing performed by the display device 1 of this embodiment. Figures 5, 6, 7, and 8 are all continuations of the previous figures. The following is an example of a preferred flow of display processing performed by the display device 1 of this embodiment, using Figures 4 to 8.

[0074] In order to display a stereoscopic image according to the viewpoint position of viewer A, the display process includes at least one of the following: a series of processes to identify the viewpoint position based on the face position identified from images acquired from camera C, a series of processes to identify the position of the left eye viewpoint and the position of the right eye viewpoint based on data related to the left eye viewpoint and the right eye viewpoint acquired from an external source, or a series of processes to identify the viewpoint position based on the face position identified from the position of infrared emitter E.

[0075] Steps S1 to S3 are an example of a series of processes for determining face position from images acquired from camera C. This allows the display device 1 to determine viewer A's face position using camera C, such as a webcam, which is less expensive than a specialized camera for determining viewer A's eye position.

[0076] Steps S4 to S6 are an example of a series of processes for determining the face position from the position of the infrared emitter E. As a result, the display device 1 can determine the face position of viewer A using the infrared emitter E and infrared sensor, which are less expensive than a special imaging device that determines the position of viewer A's eyes.

[0077] [Step S1: Determine whether to identify or determine the face position from the image] The control unit 11, in cooperation with the storage unit 13 and the communication unit 14, performs a process to identify or determine the face position from the image (first face position identification / determination step). If the control unit 11 determines that the face position has been identified, it moves the process to step S2; otherwise, it moves the process to step S4.

[0078] In the face position identification step, the control unit 11 performs the above-mentioned identification by, for example, determining whether to identify the face position from the image when it is time to acquire data related to face position identification from camera C. This data is, for example, image or video data of viewer A.

[0079] [Step S2: Get the image] The control unit 11 works in cooperation with the storage unit 13 and the communication unit 14 to execute the image acquisition unit 111. The control unit 11 then uses the image acquisition unit 111 to acquire an image or video of viewer A from camera C (image acquisition step). The control unit 11 then moves the process to step S3.

[0080] [Step S3: Identify the face position from the image] The control unit 11 works in cooperation with the storage unit 13 and the communication unit 14 to execute the face position identification unit 113. The control unit 11 then uses the face position identification unit 113 to recognize the face of viewer A, but performs image recognition processing that does not recognize the eyes of viewer A, thereby determining the face position from the aforementioned image (first face position identification step). The control unit 11 then moves the process to step S4.

[0081] [Step S4: Determine whether to identify or determine the face position from the light source position] The control unit 11, in cooperation with the storage unit 13 and the communication unit 14, performs a process to determine whether to identify or determine the face position from the light-emitting object position (second face position identification / determination step). If the control unit 11 determines that the face position can be identified, it moves the process to step S5; otherwise, it moves the process to step S7.

[0082] [Step S5: Obtain the position of the light source] The control unit 11 works in cooperation with the memory unit 13 and the communication unit 14 to execute the light-emitting element position acquisition unit 112. The control unit 11 then uses the light-emitting element position acquisition unit 112 to perform the process of acquiring the position (light-emitting element position) of the infrared light-emitting element E attached to the stereoscopic glasses G with a handle H (light-emitting element position acquisition step). The control unit 11 then moves the process to step S6.

[0083] In the light emitter position acquisition step, the light emitter position acquisition unit 112 acquires the position of the infrared light emitter E by known procedures, such as a procedure for directly acquiring the position of the infrared light emitter E from an optical sensor that detects the infrared light emitter E, or a procedure for acquiring the position of the infrared light emitter E by image recognition processing on an infrared image acquired from an optical sensor.

[0084] [Step S6: Identify face position from light source position] The control unit 11 works in cooperation with the storage unit 13 and the communication unit 14 to execute the face position identification unit 113. Then, the control unit 11 performs a process to identify the face position based on the light-emitting body position described above (second face position identification step). The control unit 11 then moves the process to step S7.

[0085] After identifying the face position, the display device 1 determines the viewpoint position based on the face position and performs a series of processes to generate an image of the virtual space as seen from that viewpoint position. Steps S7 to S10 are an example of this process.

[0086] Furthermore, in the series of processes for generating and displaying a video of a virtual space as seen from a specified viewpoint, it is preferable that the video display unit 116 displays a virtual space including terrain or buildings generated based on an image taken from above (for example, an image taken from an aircraft, or a satellite image) in 3D on a substantially horizontal display screen. This allows the display device 1 to provide an excellent virtual space experience, such as strolling through a diorama, where the user can observe urban buildings or natural terrain in detail while moving their viewpoint. The virtual space is, for example, a virtual space based on Google Earth®. In order to provide an experience closer to a diorama, in this embodiment, it is preferable that the angle between the normal of the display screen and the vertical line in the virtual space is 15 degrees or less, more preferably 5 degrees or less, and even more preferably 2 degrees or less.

[0087] [Step S7: Identify the left eye's viewpoint] The control unit 11 works in cooperation with the memory unit 13 and the communication unit 14 to execute the viewpoint identification unit 114. Then, the control unit 11 performs the process of identifying the left eye viewpoint based on the face position described above (left eye viewpoint identification step). The control unit 11 then moves the process to step S8.

[0088] In the left eye viewpoint identification step, the viewpoint identification unit 114 identifies the left eye viewpoint, for example, by adding a given offset related to the left eye viewpoint to the face position. In order to perform a stereoscopic display tailored to the individual, it is preferable that this offset can be changed by the operation of viewer A.

[0089] When display D is configured as a multi-display, it is preferable that the viewpoint identification unit 114 identifies the left-eye viewpoint for each of the multiple displays D in the left-eye viewpoint identification step. In this case, it is preferable that the offset can be changed for each of the multiple displays D.

[0090] To reduce the computational cost associated with viewpoint identification, the left eye viewpoint identification step preferably includes a procedure for identifying the position of the left eye viewpoint based on data related to the left eye viewpoint acquired from an external source. This data may include, for example, data acquired from an external device that acquires eye position, such as the VIVE Tracker®, or data acquired from various software that acquires eye position from video footage captured by camera C. To achieve both reduced computational cost and high accuracy, it is preferable that this data includes data acquired from the VIVE Tracker.

[0091] To determine the viewpoint position in the depth direction, the left eye viewpoint identification step preferably includes a procedure to determine the distance from an external device or camera C to the left eye viewpoint using the interpupillary distance in the acquired data and the standard interpupillary distance in humans. A given value, such as 65 millimeters, is used as the standard interpupillary distance. In this case, the step includes, for example, calculating the angle of the face as seen from camera C, calculating how many millimeters the standard interpupillary distance (interpupillary distance, e.g., 65 millimeters) appears to be from the camera direction at that angle, and then calculating the distance from camera C to the left eye viewpoint based on that.

[0092] [Step S8: Generate left eye image in virtual space] The control unit 11 works in cooperation with the memory unit 13 and the communication unit 14 to execute the image generation unit 115. The control unit 11 then uses the image generation unit 115 to generate a left-eye image of the virtual space as seen from the left eye's perspective (left-eye image generation step). The control unit 11 then moves the process to step S9.

[0093] In the left-eye image generation step, the image generation unit 115 generates a left-eye image by, for example, a procedure that synthesizes images of each three-dimensional model viewed from the left-eye perspective.

[0094] When the background is a live-action image or live-action video, it is preferable that in the left-eye video generation step, the video generation unit 115 generates the left-eye video by a process that includes a step of combining a background image B, which is a texture of a live-action background pasted on the inside of a hollow sphere surrounding the virtual space and viewed from the left-eye viewpoint towards the display screen, with a foreground image F, which is a three-dimensional model placed inside the hollow sphere and viewed from the left-eye viewpoint. This procedure reduces the computational load required for rendering a background similar to live-action footage and contributes to providing an excellent virtual space experience at a low cost.

[0095] When display D is configured as a multi-display, it is preferable that the image generation unit 115 generates left-eye images for each of the multiple displays D in the left-eye image generation step.

[0096] In order to perform immersive 3D display at a real scale, if the background image B is based on an image of a real landscape, it is preferable that the image generation unit 115 generates a left-eye image of the virtual space at the same scale as the real space during the left-eye image generation step.

[0097] [Step S9: Identify the right eye's viewpoint] The control unit 11 works in cooperation with the memory unit 13 and the communication unit 14 to execute the viewpoint identification unit 114. Then, the control unit 11 performs the process of identifying the right eye viewpoint based on the face position described above (right eye viewpoint identification step). The control unit 11 then moves the process to step S10.

[0098] In the right-eye viewpoint identification step, the viewpoint identification unit 114 identifies the right-eye viewpoint by, for example, adding a given offset related to the right-eye viewpoint to the face position. In order to perform a stereoscopic display tailored to the individual, it is preferable that the offset can be changed by the viewer A.

[0099] When display D is configured as a multi-display, it is preferable that the viewpoint identification unit 114 identifies the right-eye viewpoint for each of the multiple displays D in the right-eye viewpoint identification step. In this case, it is preferable that the offset can be changed for each of the multiple displays D.

[0100] To reduce the computational cost associated with viewpoint identification, the right-eye viewpoint identification step preferably includes a procedure for identifying the position of the right-eye viewpoint based on data related to the right-eye viewpoint acquired from an external source. This data may include, for example, data acquired from an external device that acquires eye position, such as the VIVE Tracker®, or data acquired from various software that acquires eye position. To achieve both reduced computational cost and high accuracy, it is preferable that this data includes data acquired from the VIVE Tracker.

[0101] To determine the viewpoint position in the depth direction, the right-eye viewpoint identification step preferably includes a procedure to determine the distance from the external device or camera C to the left-eye viewpoint using the distance between the eyes in the acquired data and the standard distance between the eyes in humans. A given value, such as 65 millimeters, is used as the standard distance between the eyes.

[0102] [Step S10: Generate the right eye image in the virtual space] The control unit 11 works in cooperation with the memory unit 13 and the communication unit 14 to execute the image generation unit 115. The control unit 11 then uses the image generation unit 115 to generate a right-eye image of the virtual space as seen from the right eye's perspective (right-eye image generation step). The control unit 11 then moves the process to step S11.

[0103] In the right eye image generation step, the image generation unit 115 generates the right eye image by a process that includes the same procedure as in the left eye image generation step.

[0104] When the display D is configured as a multi-display, it is preferable that the image generation unit 115 generates a right-eye image for each of the multiple displays D in the right-eye image generation step.

[0105] In order to perform immersive 3D display at a real scale, if the background image B is based on an image of a real landscape, it is preferable that the image generation unit 115 generates a right-eye image of the virtual space at the same scale as the real space in the right-eye image generation step.

[0106] The display device 1 performs a series of processes to perform stereoscopic display based on the left-eye image generated in the left-eye image generation step and the right-eye image generated in the right-eye image generation step. In this process, the image display unit 116 of the display device 1 displays the virtual space in stereoscopic form in one or more modes specified by the user from a group that includes at least the following: stereoscopic viewing through active shutter glasses, stereoscopic viewing through anaglyph glasses, and stereoscopic viewing with polarized glasses or the naked eye.

[0107] Steps S11 to S13 are an example of the process when the stereoscopic glasses G are anaglyph glasses. Steps S14 to S15 are an example of the process when the stereoscopic glasses G are active shutter glasses. Steps S16 to S17 are an example of the process when the stereoscopic glasses G are polarized glasses, or when the image is displayed on a display D that allows for stereoscopic viewing with the naked eye.

[0108] [Step S11: Determine whether to display the anaglyph] The control unit 11 works in cooperation with the storage unit 13 and the communication unit 14 to perform a process to determine whether to display anaglyphs (anaglyph display determination step). If the control unit 11 determines to display them, it moves the process to step S12; otherwise, it moves the process to step S14. The control unit 11 achieves the above determination by, for example, determining whether to display anaglyphs when viewer A has specified anaglyph display.

[0109] [Step S12: Generate anaglyph] The control unit 11 works in cooperation with the storage unit 13 and the communication unit 14 to execute the video display unit 116. The control unit 11 then uses the video display unit 116 to perform the process of generating an anaglyph based on the left-eye image and the right-eye image described above (anaglyph generation step). The control unit 11 then moves the process to step S13.

[0110] When display D is configured as a multi-display, it is preferable that the video display unit 116 generates an anaglyph for each of the multiple displays D in the anaglyph generation step.

[0111] In the anaglyph generation step, the video display unit 116 generates an anaglyph by a process including, for example, the following steps 1 to 4.

[0112] (Step 1) Of the red, green, and blue channels that make up the left eye image, the channel that makes up the color that is easily transmitted through the left eye lens of the anaglyph glasses (for example, the red channel) is retained as the left eye channel.

[0113] (Step 2) Of the red, green, and blue channels that make up the right eye image, the channels that make up colors that easily pass through the right-eye lens of the anaglyph glasses (for example, the green and blue channels) are retained as the right-eye channels.

[0114] (Step 3) An anaglyph is generated by appropriately overlaying the left-eye channel and the right-eye channel. In this overlay, it is preferable to overlay the channels so that the points of focus identified from the foreground image F coincide, in order to suppress the occurrence of ghost images.

[0115] (Step 4) If necessary, color tone and brightness are corrected to suppress crosstalk and ghosting.

[0116] (Regarding the example procedure) In the anaglyph generation step, the procedure by which the video display unit 116 generates the anaglyph is not limited to the process including steps 1 to 4 described above, but may be any appropriate procedure selected from various conventionally known methods. It is understood that the channel colors in steps 1 to 4 are changed as appropriate according to the color of the lenses of the anaglyph glasses.

[0117] [Step S13: Display the anaglyph] The control unit 11 executes the process of displaying the anaglyph described above using the video display unit 116 (anaglyph video display step). That is, in this step, the video display unit 116 displays the virtual space described above in an anaglyph form on the display screen (display D) in 3D. The control unit 11 then moves the process to step S14.

[0118] When display D is configured as a multi-display, it is preferable that the video display unit 116 displays the virtual space in 3D on each of the multiple displays D during the anaglyph image display step.

[0119] [Step S14: Determine whether to display for active shutter type] The control unit 11 works in cooperation with the storage unit 13 and the communication unit 14 to perform a process to determine whether to display for the active shutter type (active shutter display determination step). If the control unit 11 determines to display, it moves the process to step S15; otherwise, it moves the process to step S16. The control unit 11 achieves the above determination by, for example, determining to display for the active shutter type when viewer A has specified a display for the active shutter type glasses.

[0120] [Step S15: Display for active shutter type] The control unit 11 executes a process to display the left-eye image and the right-eye image described above in an active shutter format using the image display unit 116 (active shutter image display step). That is, in this step, the image display unit 116 displays the virtual space described above in a three-dimensional format for an active shutter format on the display screen (display D). The control unit 11 then moves the process to step S16.

[0121] In the active shutter type image display step, the image display unit 116 displays the left eye image on the display screen when the shutter of the left eye lens of the active shutter type glasses, whose shutter opening and closing timings have been pre-synchronized, opens, and displays the right eye image on the display screen when the shutter of the right eye lens opens, thereby displaying the aforementioned virtual space in 3D on the display screen (display D).

[0122] When display D is configured as a multi-display, it is preferable that the video display unit 116 displays the virtual space in 3D on each of the multiple displays D during the active shutter type video display step.

[0123] [Step S16: Determine whether to display for polarized sunglasses, etc.] The control unit 11 works in cooperation with the storage unit 13 and the communication unit 14 to perform a process to determine whether to display for polarized glasses, etc. (polarized glasses display determination step). If the control unit 11 determines to display, it moves the process to step S17; otherwise, it moves the process to step S18. "Display for polarized glasses, etc." here refers to displaying on a 3D television that can simultaneously display left-eye and right-eye images, such as a 3D television that uses polarized glasses, or a 3D television that allows for naked-eye stereoscopic viewing. The control unit 11 achieves the above determination by, for example, determining whether to display for polarized glasses, etc. when viewer A has specified this.

[0124] [Step S17: Display for polarized sunglasses, etc.] The control unit 11 executes a process to display the left-eye image and the right-eye image for polarized glasses, etc., using the image display unit 116 (image display step for polarized glasses, etc.). That is, in this step, the image display unit 116 displays the virtual space described above in 3D on the display screen (display D) in a manner suitable for polarized glasses, etc. This step is realized by a process that includes, for example, a procedure for outputting the left-eye image and the right-eye image to the display D, which is a 3D television. The control unit 11 then moves the process to step S18.

[0125] The display processing preferably includes a series of processes that acquire touch operations on the display screen from an external device such as an operation acquisition device P, and change the viewpoint position or scale in response to these touch operations. Steps S18 to S26 are an example of such processing.

[0126] Through this series of processes, the display device 1 can provide an excellent virtual space experience in which the viewpoint can be freely changed by touching the display screen, even when displaying on a relatively inexpensive display D that does not offer touch operation, without unnecessarily increasing the introduction cost.

[0127] In particular, when the display screen is a projector, this series of processes can realize a large screen at a lower cost than a flat-panel display, and even in projectors that normally do not offer touch operation, it enables operation to freely change the viewpoint by touching the display screen. As a result, the display device 1 can provide an excellent virtual space experience at a low cost.

[0128] Steps S18 to S26 are an example of processing in an operation system that assigns horizontal movement of the viewpoint to one-finger operation, rotation and scaling of the viewpoint to two-finger operation, and vertical movement of the viewpoint to three-finger operation. This example provides an intuitive operation system that switches between horizontal movement of the viewpoint, rotation and scaling of the viewpoint, and vertical movement of the viewpoint based on the number of fingers used in the touch operation, thereby providing an excellent virtual space experience without introducing any further devices to switch between types of operations.

[0129] [Step S18: Determine if a touch operation with one finger was obtained] The control unit 11 works in cooperation with the storage unit 13 and the communication unit 14 to execute the operation acquisition unit 117. The control unit 11 then performs a process to determine whether a one-finger touch operation has been acquired from an external device by the operation acquisition unit 117 (first operation acquisition step). If the control unit 11 determines that an operation has been acquired, it moves the process to step S19; otherwise, it moves the process to step S20.

[0130] [Step S19: Move the viewpoint horizontally] The control unit 11, using the viewpoint identification unit 114, performs a process to horizontally move the left eye viewpoint and the right eye viewpoint in response to the movement of the single finger (horizontal movement step). The control unit 11 then moves the process to step S20.

[0131] When the video display unit 116 displays a virtual space including terrain or buildings generated based on images taken from above in 3D on a substantially horizontal display screen, it is preferable that the viewpoint identification unit 114 moves while maintaining the elevation of the left eye viewpoint and the right eye viewpoint in response to the movement of one finger. In a virtual space including terrain or buildings generated based on satellite images, the ground can be part of a sphere instead of a plane. By moving while maintaining elevation, the display device 1 can provide intuitive movement in a virtual space where the ground is spherical.

[0132] The speed of horizontal movement preferably changes according to the scale of the virtual space in the left-eye image and the right-eye image. That is, the speed of horizontal movement preferably slows down when the scale is large and fast when the scale is small. This allows the display device 1 to provide viewer A with an excellent virtual space experience, giving them a virtual space experience from the perspective of a dwarf when the scale is large and a virtual space experience from the perspective of a giant when the scale is small.

[0133] [Step S20: Determine if a three-finger touch operation was detected] The control unit 11 performs a process to determine whether a three-finger touch operation has been acquired from an external device by the operation acquisition unit 117 (second operation acquisition step). If the control unit 11 determines that an operation has been acquired, it moves the process to step S21; otherwise, it moves the process to step S22.

[0134] [Step S21: Vertically move the viewpoint] The control unit 11, using the viewpoint identification unit 114, performs a process to vertically move the left eye viewpoint and the right eye viewpoint in accordance with the movement of the three fingers (vertical movement step). The control unit 11 then moves the process to step S22.

[0135] The speed of vertical movement preferably changes according to the scale of the virtual space in the left-eye and right-eye images. That is, the speed of vertical movement is preferably faster when rendering objects in the virtual space smaller, and slower when rendering them larger.

[0136] As a result, the display device 1 can provide viewer A with an excellent virtual space experience, giving them a virtual space experience from the perspective of a dwarf when the scale is large, and a virtual space experience from the perspective of a giant when the scale is small.

[0137] When the video display unit 116 displays a virtual space including terrain or buildings generated based on images taken from above in 3D on a substantially horizontal display screen, it is preferable that the viewpoint identification unit 114 changes the elevation of the left eye viewpoint and the right eye viewpoint in accordance with the movement of the three fingers. In a virtual space including terrain or buildings generated based on satellite images, the ground can be part of a sphere instead of a plane. By changing the elevation instead of vertical movement, the display device 1 can provide intuitive movement in a virtual space where the ground is spherical. It is also preferable to change the movement speed according to the scale, even when changing the elevation.

[0138] [Step S22: Determine if a two-finger touch operation was detected] The control unit 11 performs a process to determine whether a two-finger touch operation has been acquired from an external device by the operation acquisition unit 117 (third operation acquisition step). If the control unit 11 determines that an operation has been acquired, it moves the process to step S23; otherwise, it moves the process to step S1, and repeats the process from step S1 to step S26.

[0139] [Step S23: Determine if the direction of the line segment formed by two fingers has changed.] The control unit 11, using the operation acquisition unit 117, performs a process to determine whether the direction of the line segment with the two fingers as its ends has changed (direction change determination step). If the control unit 11 determines that the direction has changed, it moves the process to step S24; otherwise, it moves the process to step S25.

[0140] [Step S24: Rotate and move the viewpoint] The control unit 11, using the viewpoint identification unit 114, performs a process to rotate the left-eye viewpoint and the right-eye viewpoint in accordance with the change in the direction of the line segment described above (rotational movement step). That is, the viewpoint identification unit 114 rotates the direction of the line segment with the left-eye viewpoint and the right-eye viewpoint as its ends, in accordance with the change in the direction of the line segment. The control unit 11 then moves the process to step S25.

[0141] [Step S25: Determine if the length of the line segment formed by two fingers has changed.] The control unit 11, using the operation acquisition unit 117, performs a process to determine whether the length of the line segment with the two fingers as its ends has changed (length change determination step). If the control unit 11 determines that the length has changed, it moves the process to step S26; otherwise, it moves the process to step S1, and repeats the process from step S1 to step S26.

[0142] [Step S26: Change scale] The control unit 11, using the viewpoint identification unit 114, performs a process to change the scale of the virtual space in accordance with the change in the length of the line segment described above (scale change step). The control unit 11 then moves the process to step S1 and repeats the processes from step S1 to step S26.

[0143] [Alignment Step] The display device 1 preferably further performs an alignment step to align the face position and the left and right eye positions. The alignment step is implemented, for example, by having viewer A input the offset between the position of the infrared emitter E and the left and right eye positions, or by having viewer A input the offset between the face position identified based on an image acquired from camera C and the left and right eye positions. Furthermore, the alignment step preferably includes a procedure for adjusting the field of view of camera C.

[0144] The alignment step preferably includes a procedure for determining the position of camera C relative to display D. This determination may be, for example, a method using a given positional relationship that is an approximate positional relationship over a wide range of display D, a method using a given positional relationship corresponding to the size of display D, or a method for obtaining a positional relationship specified by viewer A. In the method for obtaining a positional relationship specified by viewer A, for example, a procedure may be used in which viewer A is provided with a user interface for inputting position and angle, and input is obtained through operations via the user interface.

[0145] [Effects of display processing] The following is a description of an example of the effects brought about by the display processing of this embodiment. The effects brought about by the display processing of this embodiment are not limited to the following example.

[0146] If the display device is limited to 3D televisions using active shutter or polarized glasses, as in the technology described in Patent Document 1, then users who do not own such equipment will be required to incur additional costs for introducing them.

[0147] The display processing of this embodiment can display a virtual space in 3D in one or more modes specified from the group which includes at least one mode of 3D viewing through active shutter glasses, mode of 3D viewing through anaglyph glasses, and mode of 3D viewing through polarized glasses or with the naked eye (steps S11 to S17).

[0148] This allows viewer A to enjoy a 3D virtual space using existing equipment. Furthermore, viewer A can enjoy a 3D virtual space using a standard display they already own, simply by acquiring relatively inexpensive additional equipment such as anaglyph glasses. Therefore, the display device 1 of this embodiment can provide an excellent virtual space experience at a low cost.

[0149] Furthermore, the display processing of this embodiment may include a series of processes that acquire touch operations on the display screen from an external device different from the display D (such as an operation acquisition device P), and change the viewpoint position or scale in response to these touch operations (steps S18 to S26).

[0150] Through this series of processes, the display device 1 can provide an excellent virtual space experience in which the viewpoint can be freely changed by touching the display screen, even when displaying on a relatively inexpensive display D that does not offer touch operation, without unnecessarily increasing the introduction cost.

[0151] In particular, when the display screen is a projector, this series of processes can realize a large screen at a lower cost than a flat-panel display, and even in projectors that normally do not offer touch operation, it enables operation to freely change the viewpoint by touching the display screen. As a result, the display device 1 can provide an excellent virtual space experience at a low cost.

[0152] The example from steps S18 to S26 illustrates the processing in an operation system that assigns one-finger operation to horizontal viewpoint movement, two-finger operation to viewpoint rotation and scaling, and three-finger operation to viewpoint vertical movement. This example provides an intuitive operation system that switches between horizontal viewpoint movement, viewpoint rotation and scaling, and vertical viewpoint movement based on the number of fingers used in the touch operation, thereby providing an excellent virtual space experience without unnecessarily increasing implementation costs by introducing devices to switch between operation types.

[0153] In this operating system, the display processing can take the form of changing the speed of viewpoint movement according to the scale (steps S19, S21). In this embodiment, the speed of horizontal movement is controlled to be slower when the scale is large and faster when the scale is small. As a result, the display device 1 can provide viewer A with a more intuitive and superior virtual space experience at a low cost, by giving a virtual space experience from the perspective of a dwarf when the scale is large and a virtual space experience from the perspective of a giant when the scale is small.

[0154] Furthermore, the display processing may involve the video display unit 116 displaying a virtual space, including terrain or buildings, generated based on an image taken from above, in 3D on a substantially horizontal display screen. In this embodiment, the above-described operation system may maintain the elevation of the viewpoint during horizontal movement in response to the movement of one finger, and instead of vertical movement in response to the movement of three fingers, change the elevation of the left-eye viewpoint and the right-eye viewpoint.

[0155] In a virtual space that includes terrain or structures generated based on images taken from above, the ground can be part of a sphere instead of a plane due to the curvature of the Earth. By maintaining elevation during horizontal movement and changing elevation instead of vertical movement, the display device 1 can provide intuitive movement in a virtual space where the ground is spherical at low cost.

[0156] The display processing may take the form of identifying the viewpoint based on external data (steps S7 and S9). This allows the display device 1 to process at a lower cost without having to perform relatively complex calculations related to viewpoint identification itself. Therefore, this form contributes to providing a superior virtual space experience at a low cost.

[0157] The method of identifying the viewpoint based on external data can be combined with a process that changes the viewpoint position or scale in response to touch operations on the display screen acquired from an external device (such as an operation acquisition device P) different from the display D, thereby further reducing implementation costs. In other words, this combination of method and process enables stereoscopic display using a general display D that does not support touch operations and viewpoint position acquisition, further reducing the implementation costs related to the virtual space experience. Furthermore, this combination of method and process makes it possible to lower the minimum processing power required of the display device 1 that performs the display processing in order to realize viewpoint identification and touch operation acquisition externally, while simultaneously improving the accuracy of viewpoint identification and touch operation. Therefore, this combination contributes to providing a superior virtual space experience at a low cost.

[0158] As in the technology described in Patent Document 1, when detecting the observation position based on the positions of the right and left eyes in real space calculated by the eye position calculation function of the head tracking unit, calculations are required to determine the position of the left eye viewpoint and the position of the right eye viewpoint.

[0159] On the other hand, the display device 1 that performs the above-described display processing may identify the face position (steps S1 to S3 or S4 to S6) and, based on this face position, identify the position of the left eye viewpoint and the position of the right eye viewpoint (steps S7, S9). This allows the display device 1 to identify the viewpoint position with less computation than when using the eye position calculation function.

[0160] This procedure reduces the computational complexity required to determine the positions of the left and right eye viewpoints used to generate the virtual space image. As a result, this process, in synergy with a series of processes that combine the background image B, which uses live footage generated in a procedure described later, with the foreground image F, reduces the computational power required by the computer constituting the display device 1. Therefore, this process contributes to determining the viewpoint position using low-cost equipment. Thus, this embodiment contributes to providing an excellent virtual space experience at a low cost.

[0161] The method of identifying the viewpoint based on the identified face position can be combined with a process that changes the viewpoint position or scale in response to touch operations on the display screen acquired from an external device (such as an operation acquisition device P) different from the display D. This can reduce the processing load required for display processing and improve the responsiveness of viewpoint changes. Furthermore, this combination of the method and the process can reduce the minimum processing power required of the display device 1 that performs the display processing in order to reduce the processing load of viewpoint identification and touch operation acquisition in the display device. Therefore, this combination contributes to providing a superior virtual space experience at a low cost.

[0162] The display device 1, which performs the above-described display processing, may adopt a configuration in which, for a 3D object corresponding to the background, it composites a background image B, viewed from the viewpoint towards the display screen, with a background texture based on a real photograph pasted inside a hollow sphere surrounding the virtual space, with the foreground image F (steps S8 and S10). This procedure reduces the computational amount required for rendering the background compared to a method that unfolds each of the numerous 3D objects to be rendered into the virtual space.

[0163] Therefore, the display processing described above contributes to generating virtual space images with low-cost equipment and providing an excellent virtual space experience at a low cost.

[0164] The method of compositing a background image B, generated by applying a background texture based on live-action footage to the inside of a hollow sphere, with a foreground image F, can reduce the processing load required for display processing and improve the responsiveness of viewpoint changes when combined with a process that changes the viewpoint position or scale in response to touch operations on the display screen acquired from an external device (such as an operation acquisition device P) different from the display D. Furthermore, this combination of the method and the process makes it possible to lower the minimum processing power required of the display device 1 that performs the display processing in order to reduce the processing load of rendering live-action footage and acquiring touch operations on the display device. Therefore, this combination contributes to providing an excellent virtual space experience at a low cost.

[0165] In the display process described above, the display device 1 may take the form of detecting an infrared emitter E attached to stereoscopic glasses G with a handle H (steps S4 to S6). This form can further reduce the amount of computation required to determine the position of the viewpoint compared to the process of determining the position of the eyes by an eye position calculation function (Patent Document 1) and the process of determining the position of the face by image recognition (steps S1 to S3).

[0166] As a result, this process, through a synergistic effect with a series of processes such as compositing the background image B generated using a hollow sphere with the foreground image F, further reduces the computing power required for the computer constituting the display device 1. Therefore, this process contributes to determining the position of the viewpoint with low-cost equipment. Consequently, this embodiment contributes to providing an excellent virtual space experience at a low cost.

[0167] The method of detecting an infrared emitter E attached to stereoscopic glasses G with a handle H can be further reduced in implementation costs by combining it with a process that changes the viewpoint position or scale in response to touch operations on the display screen acquired from an external device (such as an operation acquisition device P) different from the display D. In other words, this combination of method and process enables stereoscopic display by adding an operation acquisition device P and a relatively inexpensive infrared detector to a general display D that does not support touch operations and viewpoint position acquisition, further reducing the implementation costs related to the virtual space experience. Furthermore, this combination of method and process reduces the processing load of viewpoint position identification and touch operation acquisition, making it possible to lower the minimum processing power required of the display device 1 that performs the display processing. Therefore, this combination contributes to providing an excellent virtual space experience at a low cost.

[0168] By surrounding viewer A with multiple displays D and displaying a virtual space in 3D on these displays, viewer A can experience an immersive 3D viewing experience as if they had entered the virtual space.

[0169] In contrast, the Cave automatic virtual environment (CAVE), a conventional example of immersive stereoscopic display technology, requires expensive equipment such as a 3D display capable of stereoscopic viewing without glasses, a large number of LEDs that turn an entire wall or other surface into a display, a large LCD display, and a high-performance GPU.

[0170] On the other hand, the display processing of this embodiment can realize a method of displaying a virtual space in 3D on multiple displays using relatively inexpensive equipment such as a commercially available webcam, display, and personal computer (steps S7 to S15). Therefore, this processing contributes to providing an immersive and excellent virtual space experience at a low cost.

[0171] Incidentally, intuitive operation via touch controls enhances the virtual space experience. However, when configuring a multi-screen setup with large flat-screen displays that offer touch functionality, the implementation costs are high because such large flat-screen displays are relatively expensive.

[0172] As described above, the display processing of this embodiment may include a series of processes that acquire touch operations on the display screen from an external device (such as an operation acquisition device P) different from the display D, and change the viewpoint position or scale in response to these touch operations. In the case of a multi-screen setup, this embodiment allows for intuitive operation via touch to be added to an immersive 3D display experience that makes the user feel as if they have entered a virtual space, without unnecessarily increasing the cost of introducing multiple large flat displays that provide touch operations. Therefore, this embodiment contributes to providing an even better virtual space experience at a lower cost.

[0173] In particular, when the multi-display system consists of multiple screens using projectors and screens, the display processing of this embodiment can be realized with inexpensive equipment that does not require large 3D displays, a large number of LEDs to turn an entire wall into a display, or large liquid crystal displays. Therefore, this processing contributes to providing an immersive and excellent virtual space experience at a low cost.

[0174] The multi-screen configuration using a projector, etc., can be further reduced in cost by combining it with a process that changes the viewpoint position or scale in response to touch operations on the display screen acquired from an external device (such as an operation acquisition device P) different from the display D. In other words, this configuration and the process combine a projector, which can realize a large screen at a lower cost than a large flat-panel display, with an operation acquisition device P to achieve stereoscopic display, further reducing the cost of introducing an immersive virtual space experience. Therefore, this combination contributes to providing a superior virtual space experience at a low cost.

[0175] In particular, the combination of screen display and anaglyph, as described later, is expected to reduce the visibility of color shifts in anaglyph display and suppress the occurrence of crosstalk and ghost images due to the characteristic of screen display where the image on the screen is blurred compared to the image on the liquid crystal display. Therefore, this combination contributes to providing an immersive and superior virtual space experience at a low cost.

[0176] The display process described above can take the form of displaying the generated left-eye and right-eye images in stereoscopic form using anaglyphs (steps S11 to S13). This form reduces the cost of procuring stereoscopic glasses G compared to using active shutter glasses or the like as stereoscopic glasses G. Therefore, this process contributes to providing a superior virtual space experience at a low cost.

[0177] The method of displaying in 3D using anaglyphs can be combined with a process that changes the viewpoint position or scale in response to touch operations on the display screen acquired from an external device (such as an operation acquisition device P) different from the display D, thereby further reducing implementation costs. In other words, this combination of method and process enables 3D display by adding a red-blue glasses (which are less expensive than 3D TVs that allow stereoscopic viewing with the naked eye, polarized glasses, or active shutter glasses) and an operation acquisition device P, further reducing the implementation costs related to large-screen virtual space experiences. Therefore, this combination contributes to providing a superior virtual space experience at a low cost.

[0178] Furthermore, when color reproduction is a priority, the above-described display processing can be configured to stereoscopically display the generated left-eye and right-eye images for an active shutter system (steps S14 to S15). This configuration uses active shutter glasses G as stereoscopic glasses, which are not as low-cost as anaglyph glasses, but it enables stereoscopic display with high color reproduction using equipment that is less expensive than a display compatible with naked-eye stereoscopic viewing. Therefore, this processing contributes to identifying the viewpoint position at a low cost.

[0179] The display process described above may involve detecting an infrared emitter E attached to stereoscopic glasses G with a handle H (steps S4 to S6), and then displaying anaglyphs on a multiscreen using the eye position calculated based on this detection (steps S7 to S13).

[0180] In multi-screen setups, the computational complexity required to detect face positions on each screen increases, raising concerns about decreased accuracy in position detection. Furthermore, the increased total screen area in multi-screen configurations increases the likelihood of crosstalk and ghost image generation.

[0181] In this embodiment, the infrared emitter E reduces the increase in computational load and decrease in accuracy, and by further combining multi-screen and anaglyph display, it is expected that crosstalk and ghost image generation will be suppressed without the need for expensive equipment. Therefore, this embodiment contributes to providing a superior virtual space experience at a low cost.

[0182] Multi-screen displays are a promising method for exhibits aimed at a large, unspecified audience (A). However, when using stereoscopic glasses G, which are shaped like ordinary glasses with nose pads that rest on the nose and temples that go over the ears, to exhibit stereoscopic displays for a large, unspecified audience (A), the nose pads and temples used by other audience members (A) will come into contact with the face, where the skin is thin and susceptible to the entry of bacteria and viruses. Therefore, in such exhibits, there are concerns about hygiene issues arising from the reuse of these stereoscopic glasses G by a large, unspecified audience (A).

[0183] In this embodiment, by providing the handle H on the stereoscopic glasses G, hygiene concerns arising from the nose pads or temples coming into contact with the viewer A's face can be reduced without significantly increasing costs. Therefore, in this combination, this process further contributes to providing immersive stereoscopic display at a low cost.

[0184] The configuration in which the handle H is provided on the stereoscopic glasses G, when combined with a process that changes the viewpoint position or scale in response to touch operations on the display screen acquired from an external device (such as an operation acquisition device P) different from the display D, further reduces contact between the stereoscopic glasses G and the input device and the viewer A. Therefore, this combination can alleviate concerns about nose pads or temples coming into contact with the viewer A's face, and hygienic concerns arising from the viewer A holding an input device such as a mouse in their hand, without significantly increasing costs. Thus, this combination and the process further contribute to providing immersive stereoscopic displays at a low cost.

[0185] <Usage example> The following is an example of how to use the display device 1 of this embodiment.

[0186] [Viewing using anaglyph glasses] The following is an example of viewing using anaglyph glasses as stereoscopic glasses G.

[0187] [Preparing glasses for anaglyph Viewer A prepares anaglyph glasses as stereoscopic glasses G. Viewer A uses the anaglyph glasses via a handle H, etc.

[0188] [Alignment] Viewer A aligns the position of their face, left eye, and right eye by using appropriate input methods.

[0189] [3D display] Viewer A commands display device 1 to start the 3D display. After identifying the face position, display device 1 calculates the left eye position and right eye position based on this face position and generates the left eye image and the right eye image. Then, display device 1 displays the anaglyph on display D. Viewer A views the anaglyph displayed on display D as a 3D image through anaglyph glasses.

[0190] [Viewing using active shutter glasses] The following is an example of viewing using active shutter glasses as stereoscopic glasses G.

[0191] [Preparing the active shutter glasses] Viewer A prepares active shutter glasses G as stereoscopic viewing glasses. Viewer A puts on the active shutter glasses via a handle H or the like.

[0192] [Synchronization and alignment] Viewer A synchronizes the shutter opening and closing timing of the active shutter glasses with the display timing of the display device 1 via appropriate communication means. Subsequently, Viewer A aligns their face position, left eye position, and right eye position using appropriate input means.

[0193] [3D display] Viewer A commands display device 1 to start the 3D display. After identifying the face position, display device 1 calculates the left eye position and right eye position based on this face position and generates the left eye image and the right eye image. Then, display device 1 displays the left eye image and the right eye image on display D in a display mode that alternately switches for active shutter type. Viewer A views the left eye image and the right eye image, which are alternately displayed on display D, as a 3D image through active shutter type glasses.

[0194] Within the scope of the concept of this invention, those skilled in the art can conceive of various modifications and alterations. Therefore, such modifications and alterations are understood to fall within the scope of this invention. For example, any addition, deletion, or design change of components, or addition, omission, or modification of processes, made by a person skilled in the art to the above-described embodiments, is also included within the scope of this invention, as long as it retains the essence of this invention. [Explanation of symbols]

[0195] S System 1 Display device 11 Control Unit 111 Image acquisition unit 112 Light-emitting element position acquisition unit 113 Face position identification unit 114 Perspective Identification Section 115 Video Generation Unit 116 Video display unit 117 Operation acquisition part 13 Storage section 131 3D Model Database 14 Communications Department C Camera D Display B Background image F Foreground image A viewer G Stereoscopic Glasses E Infrared emitter H handle P operation acquisition device N Network T terminal

Claims

1. A face position identification unit that identifies the location of the viewer's face, A viewpoint identification unit that identifies the position of the left eye viewpoint and the position of the right eye viewpoint related to the virtual space displayed on the display screen based on the face position, A video generation unit that generates a left-eye image of the virtual space viewed from the left-eye perspective and a right-eye image of the virtual space viewed from the right-eye perspective, A video display unit that displays the virtual space in 3D on the display screen based on the left eye image and the right eye image, A display device comprising, The image display unit displays the virtual space in 3D in one or more modes specified by the user from the group which includes at least one mode of viewing in 3D through active shutter glasses, a mode of viewing in 3D through anaglyph glasses, and a mode of viewing in 3D with polarized glasses or the naked eye. moreover, The aforementioned video display unit displays the virtual space in 3D on the display screen projected by the projector. The display device further includes an operation acquisition unit that acquires touch operations on the display screen from an external device different from the projector, The aforementioned viewpoint identification unit is In the case of a single-finger touch operation, the left eye view and the right eye view are moved horizontally in accordance with the movement of the finger. In the case of touch operation using two fingers, the orientation of the line segments with the left eye viewpoint and the right eye viewpoint as ends, and the scale of the virtual space are changed in accordance with the change in the orientation and length of the line segments with the two fingers as ends. In the case of touch operation using three fingers, the left eye view and the right eye view are moved vertically in accordance with the movement of the three fingers. A display device for virtual space.

2. The display device according to claim 1, wherein the viewpoint identification unit changes the speed of the horizontal movement and the vertical movement according to the scale.

3. The aforementioned video display unit displays a virtual space, including terrain or buildings, generated based on images taken from above, in 3D on a substantially horizontal display screen. The aforementioned viewpoint identification unit is In the case of a single-finger touch operation, the left and right eye viewpoints are moved while maintaining their elevation in accordance with the movement of that single finger. In the case of touch operation using three fingers, the elevation of the left eye viewpoint and the right eye viewpoint are changed in accordance with the movement of the three fingers. The display device according to claim 1.

4. The image generation unit generates the left-eye image and the right-eye image by a process that includes a step of combining a background image viewed from the viewpoint towards the display screen with a foreground image of a three-dimensional model placed inside the hollow sphere surrounding the virtual space, viewed from the left-eye viewpoint and the right-eye viewpoint, respectively. The display device according to any one of claims 1 to 3.

5. The viewpoint identification unit identifies the position of the left eye viewpoint and the position of the right eye viewpoint based on data related to the left eye viewpoint and the right eye viewpoint acquired from an external source. The display device according to any one of claims 1 to 3.

6. The display device according to any one of claims 1 to 3, wherein the video display unit displays the virtual space in three dimensions on a plurality of display screens arranged around the viewer.

7. The display device according to any one of claims 1 to 3, wherein the video display unit displays the virtual space in an anaglyph manner on the display screen in three dimensions.

Images