Display device, control method thereof, and program

Abstract

To enable natural movements of a user in a virtual space.SOLUTION: A display device comprises: a real object information acquisition section that acquires information on a position of a real object, which is an object in a real space where a user is actually present, in the real space, and information on a positional relation between the real object and the user in the real space; a virtual object information acquisition section that acquires information on a position of a virtual object, which is an object in a virtual space, in the virtual space; and a user position acquisition section that aligns the positions and poses of the real object and the virtual object and calculates a positional relation in the virtual space between the virtual object and the user in the virtual space on the basis of the information on the positional relation in the real space, to acquire user position information indicating the position of the user in the virtual space on the basis of the positional relation in the virtual space.SELECTED DRAWING: Figure 7

Description

Technical Field

[0001] The present invention relates to a display device, a control method for the display device, and a program.

Background Art

[0002] In recent years, information devices have evolved significantly. Such information devices include not only smartphones, which are representative examples, but also so-called wearable devices worn by users, which have become widespread. As wearable devices, there is an HMD (Head Mount Display) that provides visual information and is used for AR (Augumented Reality) and VR (Virtual Reality). For example, Patent Document 1 discloses a method of adjusting the position of an object displayed on the output unit of an HMD using a predetermined AR marker.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] When a wearable device provides a virtual space to a user, the user can freely determine their position in the virtual space. However, if the user's position relative to an object in the real space and the virtual space does not match, the user's behavior in the virtual space may become unnatural.

[0005] In view of the above problems, an object of the present invention is to provide a display device, a control method for the display device, and a program capable of suppressing the unnatural behavior of a user in a virtual space.

Means for Solving the Problems

[0006] A display device according to an aspect of the present invention includes: a real object information acquisition unit that acquires information on the position of a real object, which is an object in the real space where the user exists, and information on the positional relationship between the real object and the user in the real space; a virtual object information acquisition unit that acquires information on the position of a virtual object, which is an object in the virtual space, in the virtual space; and a user position acquisition unit that aligns the positions and postures of the real object and the virtual object, calculates the positional relationship between the virtual object and the user in the virtual space based on the information on the positional relationship in the real space, and acquires user position information indicating the position of the user in the virtual space based on the positional relationship in the virtual space.

[0007] A control method for a display device according to an aspect of the present invention includes: a step of acquiring information on the position of a real object, which is an object in the real space where the user exists, and information on the positional relationship between the real object and the user in the real space; a step of acquiring information on the position of a virtual object, which is an object in the virtual space, in the virtual space; a step of aligning the positions and postures of the real object and the virtual object, calculating the positional relationship between the virtual object and the user in the virtual space based on the information on the positional relationship in the real space, and acquiring user position information indicating the position of the user in the virtual space based on the positional relationship in the virtual space.

[0008] A program according to an aspect of the present invention causes a computer to execute steps of: acquiring information on the position of a real object, which is an object in a real space where a user exists, in the real space, and information on the positional relationship between the real object and the user in the real space; acquiring information on the position of a virtual object, which is an object in a virtual space, in the virtual space; aligning the positions and postures of the real object and the virtual object, calculating the positional relationship between the virtual object and the user in the virtual space based on the information on the positional relationship in the real space, and acquiring user position information indicating the position of the user in the virtual space based on the positional relationship in the virtual space.

Advantages of the Invention

[0009] According to the present invention, the behavior of a user in a virtual space can be made natural.

Brief Description of the Drawings

[0010]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Embodiments for Carrying Out the Invention

[0011] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. Note that the present invention is not limited to the embodiments described below.

[0012] (Real Space and Virtual Space) FIG. 1 is a schematic diagram for explaining an example of a real space and a virtual space. The display device 10 according to the present embodiment is a display device that displays an image. As shown in FIG. 1, the display device 10 is a so-called HMD (Head Mount Display) worn on the head of the user U. By displaying an image, the display device 10 provides a virtual space to the user U. As shown in FIG. 1, the real space SR in which the user U actually exists is defined as the real space, and the virtual space SV provided by the display device 10 to the user U is defined as the virtual space. In this case, the display device 10 displays an image for the virtual space SV according to the operation (line of sight) of the user U in the real space SR. That is, an image for the virtual space SV is displayed by simulating the operation of the user U as the user UV in the virtual space SV. Therefore, the user U can recognize that he / she exists in the virtual space SV. Here, the virtual space SV may be an MR (Mixed Reality), that is, a space that reproduces an actual place away from the place where the user U exists, or a virtual space that does not actually exist, that is, a VR (Virtual Reality). Hereinafter, in the coordinate system of the real space SR, one direction along the horizontal direction is defined as the direction XR, the direction orthogonal to the direction XR along the horizontal direction is defined as the direction YR, and the vertical direction is defined as the direction ZR. Also, in the coordinate system of the virtual space SV, one direction along the horizontal direction is defined as the direction XV, the direction orthogonal to the direction XV along the horizontal direction is defined as the direction YV, and the vertical direction is defined as the direction ZV.

[0013] (Display System) FIG. 2 is a schematic diagram of the display system according to the present embodiment. As shown in FIG. 2, the display system 1 according to the present embodiment includes a display device 10 and an information processing device 12. In the present embodiment, the display system 1 provides a virtual space SV to the user U. In the example of FIG. 1, the user U wears the display device 10, and the display device 10 provides an image in the virtual space SV within the visual field range V of the user UV in the virtual space SV to the user U as an image for the virtual space SV. Note that the number of avatars in the virtual space SV (that is, the number of users U to whom the common virtual space SV is provided) may be arbitrary. Also, in FIG. 2 and the subsequent figures, although the virtual space SV is represented as a two-dimensional plane viewed from directly above in the vertical direction, the virtual space SV may be a three-dimensional space.

[0014] The information processing device 12 is a device that performs information processing on information for the virtual space SV. The information processing device 12 can be said to be a server that transmits and receives information to and from the display device 10 of the user U and performs image processing on an image for the virtual space SV. For example, the information processing device 12 is a computer including an arithmetic device including an arithmetic circuit such as a CPU (Central Processing Unit) and a storage unit, and executes processing by reading and executing a program (software) from the storage unit.

[0015] (Display device) FIG. 3 is a schematic block diagram of the display device according to the present embodiment. The display device 10 can also be said to be a computer. As shown in FIG. 3, the display device 10 includes an input unit 20, a display unit 22, a storage unit 24, a communication unit 26, a real space detection unit 28, and a control unit 30. The input unit 20 is a mechanism that receives an operation by the user U, and may be, for example, a controller or a microphone provided in the HMD. The display unit 22 is a display that displays an image. The display unit 22 provides the virtual space SV to the user U by outputting an image. In addition to the display unit 22, the display device 10 may include a device that outputs information, such as a speaker that outputs sound.

[0016] The storage unit 24 is a memory that stores various information such as the calculation content and programs of the control unit 30, and includes at least one of, for example, a main storage device such as a RAM (Random Access Memory) and a ROM (Read Only Memory), and an external storage device such as an HDD (Hard Disk Drive). The program for the control unit 30 stored in the storage unit 24 may be stored in a recording medium readable by the display device 10.

[0017] The communication unit 26 is a communication module that communicates with an external device, such as an antenna. The display device 10 communicates with an external device such as the information processing device 12 by wireless communication, but may also use wired communication, and the communication method may be arbitrary.

[0018] The real space detection unit 28 is a sensor that detects the position of the user U in the real space SR, the posture of the user U, and the surroundings of the display device 10 (user U). The position of the user U refers to the position of the user U with the initial position of the user U as the origin in the XR - YR plane. The initial position of the user U is the position in the real space SR when the display device 10 is activated, that is, it can be said to be the first position of the user U when using the display device 10. The posture of the user U refers to the magnitude of the angle formed by the XR - YR plane and the direction in which the display device 10 faces (the direction in which the user U's face faces). The real space detection unit 28 detects the position of the user U by, for example, an acceleration sensor, detects the posture of the user U by a gyro sensor, and detects the surroundings of the display device 10 (user U) by a 3D camera such as a stereo camera or a ToF (Time of Flight) camera. However, the real space detection unit 28 is not limited to this configuration as long as it can detect the position of the user U in the real space SR, the posture of the user U, and the objects existing in the real space SR around the display device 10 (user U). For example, the real space detection unit 28 may detect the objects existing in the real space SR using LIDAR (Light Detection And Ranging), and may also detect the posture of the user U using the image of the real space SR captured by a camera.

[0019] The control unit 30 is an arithmetic unit and includes an arithmetic circuit such as a CPU (Central Processing Unit). The control unit 30 includes an actual object information acquisition unit 40, a virtual object information acquisition unit 42, a user position acquisition unit 44, and a provision unit 46. The control unit 30 reads and executes a program (software) from the storage unit 24 to realize the actual object information acquisition unit 40, the virtual object information acquisition unit 42, the user position acquisition unit 44, and the provision unit 46, and execute their processes. Note that the control unit 30 may execute these processes by one CPU, or may include a plurality of CPUs and execute the processes by these plurality of CPUs. Further, at least a part of the processes of the actual object information acquisition unit 40, the virtual object information acquisition unit 42, the user position acquisition unit 44, and the provision unit 46 may be realized by a hardware circuit.

[0020] (Actual Object Acquisition Unit) The actual object information acquisition unit 40 acquires information on the actual object OR. The actual object OR refers to an object existing around the display device 10 (user U) in the coordinate system of the actual space SR. The information on the actual object OR includes information indicating the positional relationship between the user U and the actual object OR in the actual space SR (relative position information of the actual object OR) and information indicating the position of the actual object OR in the actual space SR (absolute position information of the actual object OR). Further, the information on the actual object OR preferably also includes information indicating the shape of the actual object OR in the actual space SR (shape information of the actual object OR). In the present embodiment, the actual object information acquisition unit 40 controls the actual space detection unit 28 to cause the actual space detection unit 28 to detect an object around the display device 10 (user U). Then, the actual object information acquisition unit 40 extracts a plurality of feature points and regions in the detected object from the detection result by the actual space detection unit 28 by a known technique, and acquires them as information on the actual object OR based on the extraction result. However, the method of acquiring the information on the actual object OR is not limited to being detected by the actual space detection unit 28. For example, information on the position and shape of the furniture in the user U's room may be preset as information on the actual object OR, and the actual object information acquisition unit 40 may acquire the preset information on the actual object OR.

[0021] FIG. 4 is a schematic diagram showing an example of the actual space. FIG. 4 is an example of a plan view when the actual space SR is viewed from the ZR direction. The actual object information acquisition unit 40 acquires the relative position information of the actual object OR and the shape information of the actual object OR as information on the actual object OR. That is, it can be said that the actual object information acquisition unit 40 acquires, as information on the actual object OR, information indicating the position occupied by the actual object OR in the actual space SR with respect to the position of the user U.

[0022] The type of the target real object OR acquired by the real object information acquisition unit 40 may be arbitrary. However, in the present embodiment, the real object information acquisition unit 40 uses an object that does not move in the real space SR (such as a desk or a wall) as the real object OR, and acquires the position information and shape information of the object that does not move in the real space SR as the information of the real object OR. In the present embodiment, the real object information acquisition unit 40 may acquire the information of the real object OR that has a plane and cannot be moved by the user U based on the information of the real space SR. In this case, the real object information acquisition unit 40 may display the candidates of the real object OR to be acquired on the display unit 22, extract the real object OR specified by the user U through the input unit 20 from among the candidate real objects OR, and acquire the information of the extracted real object OR. However, the real object OR is not limited to being set based on the information of the real space SR. For example, the position of the real object OR, such as the layout information of the user U's room, is preset, and the real object information acquisition unit 40 may acquire the information of the position of the set real object OR.

[0023] In this embodiment, the actual object information acquisition unit 40 acquires information on the actual object OR within the actual local space LR, which is a space within a predetermined range in the actual space SR. That is, in this embodiment, an object within the range of the actual local space LR in the actual space SR is extracted as the actual object OR, and the actual object information acquisition unit 40 acquires information on the actual object OR within the actual local space LR. The actual object information acquisition unit 40 acquires the difference between the coordinates of the user U and the coordinates of the actual object OR in the actual local space LR as the relative position information of the actual object OR. The actual local space LR refers to a space within the actual space SR that is detected by the actual space detection unit 28 and is centered on the initial position of the user U. It can be said that this is the space in which the user U is assumed to act in the actual space SR. That is, for example, when the user U moves such as walking in the actual space, the coordinates of the user U in the actual local space LR change. At this time, the actual object information acquisition unit 40 updates the information on the positional relationship of the actual object OR with respect to the user U based on the coordinates of the user U in the changed actual local space LR.

[0024] Note that FIG. 4 is merely an example, and the shape and size of the actual local space LR and the actual object OR are not limited to the example in FIG. 4 and may be arbitrary.

[0025] (Virtual object information acquisition unit) The virtual object information acquisition unit 42 acquires information on the virtual object OV. The virtual object OV is a virtual object existing in the virtual space SV, and is an object to be displayed as an image for the virtual space SV in the coordinate system of the virtual space SV. The information on the virtual object OV includes information indicating the position of the virtual object OV in the virtual space SV (position information of the virtual object OV). The virtual object information acquisition unit 42 extracts a plurality of feature points and regions in the virtual object OV by a known technique, and acquires the extraction result as information on the virtual object OV. The virtual object information acquisition unit 42 may acquire the position (coordinates) of the virtual object OV with respect to the origin coordinates of the virtual space SV as the position information of the virtual object OV, or may acquire the position (coordinates) of the virtual object OV with respect to the initial setting position of the user UV in the virtual space SV as the position information of the virtual object OV. Note that the initial setting position of the user UV in the virtual space SV is a position preset as the default position of the user UV in the virtual space SV. The virtual object information acquisition unit 42 acquires information on the virtual object OV from the information processing device 12 via, for example, the communication unit 26. In the present embodiment, the virtual space SV is not constructed according to the environment around the user U in the real space SR, but is preset regardless of the environment around the user U in the real space SR. The position of the virtual object OV in the virtual space SV is also preset in the same manner as the initial setting position of the user UV.

[0026] FIG. 5 is a schematic diagram showing an example of a virtual space. FIG. 5 is an example of a plan view when the virtual space SV is viewed from the ZV direction.

[0027] Also, the information of the virtual object OV preferably includes information indicating the shape of the virtual object OV in the virtual space SV (the shape information of the virtual object OV). In the present embodiment, the virtual object information acquisition unit 42 extracts the virtual object OV based on the shape information of the real object OR and the shape information of the virtual object OV. For example, the virtual object information acquisition unit 42 extracts a virtual object OV having a shape similar to that of the real object from among the candidates of the virtual object OV in the virtual space SV, and acquires the information of the extracted virtual object OV. The method for extracting the virtual object OV similar to the real object may be arbitrary. For example, the virtual object information acquisition unit 42 selects, from among the candidates of the virtual object OV, a virtual object OV whose plane direction, height, and area have the smallest deviation from the plane direction, height, and area of the plane provided in the real object OR (for example, if the real object OR is a desk, an object having a horizontal plane in the virtual space SV and the closest height to the desk). Also, by inputting the shape information of the candidate virtual object OV into an AI (Artificial Intelligence) model capable of calculating the similarity to the real object when the shape information of the virtual object OV is input, a virtual object OV similar to the real object may be extracted. At this time, the virtual object information acquisition unit 42 may cause the display unit 22 to display the candidates of the virtual object OV to be acquired, and acquire the information of the virtual object OV selected by the user U through the input unit 20. The method for extracting the virtual object OV is not limited to this. For example, it may be extracted in advance when setting the virtual space SV, or may be extracted by the virtual object information acquisition unit 42 based on the position of the user UV in the virtual space SV, for example.

[0028] In this embodiment, the virtual object information acquisition unit 42 acquires information on a virtual object OV within a virtual local space LV, which is a space within a predetermined range in the virtual space SV. That is, in this embodiment, a virtual object within the range of the virtual local space LV in the virtual space SV is extracted as the virtual object OV, and the virtual object information acquisition unit 42 acquires information on the virtual object OV within the virtual local space LV. The virtual object information acquisition unit 42 acquires the coordinates of the virtual object OV in the virtual local space LV as the position information of the virtual object OV. Note that the virtual local space LV refers to a space corresponding to the actual local space LR in the virtual space SV, and a user UV exists at the same position as the user U in the actual local space LR. More specifically, for example, when the position and direction of the user U in the actual local space LR change, such as when the user U walks in the actual space SR (actual local space LR), the position and direction of the user UV in the virtual local space LV are similarly changed, so that the same operation as in the actual space SR is realized in the virtual space SV. The virtual local space LV is sufficiently small with respect to the virtual space SV and can be freely moved and rotated in the virtual space SV. More specifically, for example, when the user U uses the input unit 20 to input the movement of the user UV or rides on a vehicle on the virtual space SV, the position and direction of the virtual local space LV for the user UV are changed in the virtual space SV, so that the user UV can be moved within the virtual space SV while the position of the user UV in the actual space SR (actual local space LV) is fixed.

[0029] Note that FIG. 5 is merely an example, and the shapes and sizes of the virtual space SV, the virtual local space LV, and the virtual object OV are not limited to the example in FIG. 5 and may be arbitrary. Further, the virtual object information acquisition unit 42 may acquire the information on the position of the virtual object OV from the storage unit 24.

[0030] (User position acquisition unit) The user position acquisition unit 44 acquires user position information. The user position information is information indicating the position of the user UV in the virtual space SV, which is set based on the information of the real object OR acquired by the real object information acquisition unit 40 and the information of the virtual object OV acquired by the virtual object information acquisition unit 42. In this embodiment, the user position acquisition unit 44 acquires the user position information by setting the user position information by itself, but it is not limited thereto. For example, the user position information set by other devices or the user position information pre-stored in the storage unit 24 may be acquired.

[0031] In this embodiment, the user position acquisition unit 44 sets the user position information according to the positional relationship between the virtual object OV and the user UV in the virtual space SV, which is calculated based on the positional relationship between the real object OR and the user U. More specifically, the user position acquisition unit 44 sets the user position information so that the position of the virtual object OV with respect to the user UV in the virtual space SV (virtual local space LV) corresponds to the position of the real object OR with respect to the user U in the real space SR (real local space LR). In other words, the position of the user UV in the virtual space SV indicated by the user position information is set so that the direction and distance from the virtual object OV to the user UV correspond to (more preferably, match) the direction and distance from the real object OR to the user U in the real space SR. Hereinafter, a specific method for setting the user position information will be described.

[0032] FIG. 6 is a schematic diagram showing an example in which the actual local space is superimposed on the virtual space. In the present embodiment, the user position information is information for superimposing the virtual local space LV including the user UV on the virtual space SV including the virtual object OV. In other words, it is information on the position and direction of the virtual local space LV in the virtual space SV. That is, the user position information can also be said to be a value for converting the coordinate system of the actual local space LR into the coordinate system of the virtual space SV via the virtual local space LV. In the example of FIG. 6, in the virtual local space LV, one direction along the horizontal direction is defined as the direction X, the direction orthogonal to the direction X along the horizontal direction is defined as the direction Y, and the vertical direction is defined as the direction Z.

[0033] In the present embodiment, as shown in the example of FIG. 6, the user position information is set under the condition that the range corresponding to the actual object OR on the virtual local space LV overlaps the range occupied by the virtual object OV on the virtual space SV. More specifically, the user position information is set under the condition that the feature point or range on the actual object OR with respect to the user U in the actual space SR and the feature point or range of the virtual object OV with respect to the user UV in the virtual space SV overlap. In the example of FIG. 6, in the virtual space SV, an example of moving the position and orientation of the virtual local space LV is shown, but it is not limited thereto, and it may be set by moving the position and orientation of the virtual space SV while fixing the position and orientation of the virtual local space LV. Also, overlapping the feature point or range on the actual object OR with respect to the user U and the feature point or range of the virtual object OV with respect to the user UV does not necessarily mean that the feature points and ranges are exactly the same, and may refer to the feature points and ranges being approximately matched.

[0034] In this embodiment, the user position information is set based on a predetermined condition that a feature point or range of the real object OR overlaps with a feature point or range of the virtual object OV on the virtual space SV in the virtual local space LV. More specifically, the user position information is extracted as the position and orientation of the virtual local space LV in the virtual space SV that satisfy a predetermined condition, and is set as the correspondence relationship between the coordinate system of the virtual space SV and the coordinate system of the real space SR. Also, when the feature point or range of the real object OR does not match the feature point or range of the virtual object OV, the user position information is set to, for example, the one with the minimum displacement from the initial set position of the user UV in the virtual space SV among the positions and directions that satisfy a predetermined condition. The method for extracting the position and orientation of this virtual local space LV is not limited to this, and it may be set by an operation through the input unit 20 of the user U.

[0035] Also, the user position acquisition unit 44 may acquire the user position information when the user U performs an operation that satisfies a predetermined condition. That is, in this embodiment, the user position acquisition unit 44 sets the user position information triggered by the user U performing an operation that satisfies a predetermined condition. In this case, the position of the virtual local space LV is fixed within the virtual space SV, and the position of the virtual object OV with respect to the user UV in the virtual space SV is set to correspond to the position of the real object OR with respect to the user U in the real space SR. The operation of the user U that satisfies a predetermined condition may be any operation, for example, the user U performing a predetermined operation on the input unit 20, performing actions such as contacting or applying weight to a predetermined real object OR, or actions performed by the user U within the virtual space SV such as riding on a vehicle on the virtual space SV.

[0036] In the virtual space SV, when a plurality of users UV are arranged, the display device 10 of each user U may correspond the positions of virtual objects OV in a number less than the number of users to the positions of real objects OR in a plurality of real local spaces. When a plurality of users UV share one virtual object OV (for example, when a plurality of users UV on the same virtual space SV share one desk during a meeting or the like), each virtual local space LV is arranged on the virtual space SV such that the feature points or ranges of the real object OR in each virtual local space LV overlap with the feature points or ranges of the virtual object OV on the virtual space SV.

[0037] In the above description, the user position information was set under the condition that the feature points or ranges of the real object OR on the virtual local space LV and the feature points or ranges of the virtual object OV on the virtual space SV overlap two-dimensionally, but it is not limited to two dimensions. For example, the user position information may be set under the condition that the feature points or ranges of the real object OR on the virtual local space LV and the feature points or ranges of the virtual object OV on the virtual space SV overlap three-dimensionally.

[0038] (Provision unit) The provision unit 46 provides the virtual space SV to the user U based on the user position information. The provision unit 46 sets the position indicated by the user position information as the position of the user UV in the virtual space SV, and provides the virtual space SV to the user U. Specifically, the provision unit 46 acquires the information on the position and orientation of the user U in the real space SR, and converts the position and orientation of the user U in the real space SR into the position and orientation of the viewpoint of the user UV indicated by the user position information in the coordinate system of the virtual space SV. The provision unit 46 causes the display unit 22 to display, as an image for the virtual space SV, an image of the virtual space SV when viewed from the calculated position and orientation of the viewpoint of the user U.

[0039] In this way, the position and orientation of the user U in the real space SR are reflected in the position and orientation of the viewpoint of the user UV in the virtual space SV. Therefore, when the user U moves in the real space SR, the position and orientation of the viewpoint of the user UV in the virtual space SV also move. In this case, it is preferable that the amount of movement of the user U in the real space SR and the amount of movement of the viewpoint of the user UV in the virtual space SV are associated with each other. More specifically, when setting the user position information, if the size of the virtual local space LV with respect to the real local space LR is changed, it is preferable to reflect the degree of change in the size of the virtual local space LV in the amount of movement. Specifically, assuming that the ratio (scale factor) of the change in the size of the virtual local space LV with respect to the real local space LR is used as the change ratio, the providing unit 46 causes the display unit 22 to display an image for the virtual space SV as if the viewpoint of the user U has moved by an amount of movement that is the reciprocal multiple of the change ratio with respect to the amount of movement of the user U in the real space SR. That is, the providing unit 46 causes the display unit 22 to display an image of the virtual space SV from a viewpoint that has moved by an amount of movement that is the reciprocal multiple of the change ratio with respect to the amount of movement of the user U in the real space SR. For example, when the size of the virtual local space LV is doubled when setting the user position information, the providing unit 46 causes the display unit 22 to display an image of the virtual space SV from a viewpoint that has moved by an amount of movement that is 1 / 2 times the amount of movement of the user U in the real space SR.

[0040] Here, the providing unit 46 may display an image indicating the real object OR in the virtual space SV so as to overlap the virtual object OV. The providing unit 46 determines whether to display an image indicating the real object OR in the virtual space SV based on the shape information of the virtual object OV and the shape information of the real object OR. If it is determined to display, the providing unit 46 may display an image indicating the real object OR in the virtual space SV so as to overlap the virtual object OV. For example, when the size of the virtual object OV is larger than the size of the real object OR, the providing unit 46 may display an image indicating the real object OR in the virtual space SV so as to overlap the virtual object OV. In this case, the display unit 22 may provide AR (Augumented Reality) that displays an image of the virtual space SV while transmitting the real space SR, or may display the image of the virtual space SV and the image indicating the real object OR in an overlapping manner. Also, for the portion of the virtual object OV that does not overlap the real object OR, it may be deleted from the image of the virtual space SV or presented as information in an area not associated with the real object OR.

[0041] (Flowchart) FIG. 7 is a flowchart according to the present embodiment. When the user U performs a predetermined operation (step S10: Yes), the real object information acquisition unit 40 acquires information on the real object OR through the real space detection unit 28 (step S12). The virtual object information acquisition unit 42 acquires information on the virtual object OV from the information processing device 12 (step S14). Then, the user position acquisition unit 44 sets user position information based on the information on the real object OR and the information on the virtual object OV (step S16). The providing unit 46 sets the position indicated by the user position information as the position of the user UV in the virtual space SV and provides the virtual space SV to the user U (step S20). On the other hand, when the user U does not perform a predetermined operation (step S10: No), the user position acquisition unit 44 sets the initial setting position as the position of the user UV in the virtual space SV and provides the virtual space SV to the user U (step S18).

[0042] In addition, in this embodiment, although the display device 10 used by the user U sets the user position information, the entity that sets the user position information is not limited to the display device 10, and it may be performed by the information processing device 12. In this case, the information processing device 12 acquires the information of the real object OR from the display device 10, and sets the user position information from the information of the real object OR and the information of the virtual object OV set by itself. The user position acquisition unit 44 of the display device 10 acquires the user position information from the information processing device 12.

[0043] Here, in the virtual space SV, the movement of the user U in the real space SR is reflected in the movement of the user UV in the virtual space SV. However, for example, when the virtual space SV is set regardless of the layout in the real space SR, the movement of the user U in the virtual space SV may become unnatural. For example, when the user U sits in front of a desk in the real space SR, if there is no desk at that location in the virtual space SV, the user UV may be visually recognized by other users in the virtual space SV as sitting in a place where there is nothing. On the other hand, according to this embodiment, by appropriately setting the position of the user UV in the virtual space SV, the position of the real object OR with respect to the user U in the real space SR and the position of the virtual object OV with respect to the user UV in the virtual space SV can be made to correspond. As a result, the behavior of the user UV in the virtual space SV (for example, the behavior of the user UV sitting in front of the desk which is the virtual object OV) can be made to look natural. Also, thereby, the user U in the real space SR can perform an action of handling the virtual object OV in the same way as the real object OR (for example, an action of placing an object in the real space SR while visually recognizing the desk which is the virtual object OV), so that the behavior of the user U in the virtual space SV can be made natural.

[0044] (Another example of the method for setting user position information) In this embodiment, it has been described that the user position information is set by adjusting the position and orientation of the virtual local space LV in the virtual space SV based on the difference between the coordinates of the user U and the coordinates of the real object OR in the actual local space LR and the coordinates of the virtual object OV in the virtual space SV. However, the method for setting the user position information is not limited to this. Other examples will be described below.

[0045] For example, the real object information acquisition unit 40 may acquire the relative coordinates of the real object OR with the user U as the origin as information on the positional relationship of the real object OR with respect to the user U. In this case, when the user U moves in the real space SR, the relative coordinates of the real object OR will change.

[0046] Also, the virtual object information acquisition unit 42 may acquire the relative coordinates of the virtual object OV with the user UV as the origin as information on the position of the virtual object OV. In this case, when the user UV moves in the virtual space SV, the relative coordinates of the virtual object OV will change.

[0047] When the relative coordinates of the real object OR with the user U as the origin are acquired, the user position information may be set by directly corresponding the relative coordinates of the virtual object OV with the user UV as the origin to the relative coordinates of the real object OR with the user U as the origin.

[0048] (Effects of the present invention) The display device, the control method of the display device, and the program described in each embodiment can be understood as follows, for example.

[0049] The display device according to the first aspect includes a real object information acquisition unit 40 that acquires information on the position of a real object OR, which is an object in the real space SR where the user U exists, in the real space SR, and information on the positional relationship between the real object OR and the user U in the real space SR, a virtual object information acquisition unit 42 that acquires information on the position of a virtual object OV, which is an object in the virtual space SV, in the virtual space SV, and a user position acquisition unit 44 that aligns the positions and postures of the real object OR and the virtual object OV, calculates the positional relationship between the virtual object OV and the user UV in the virtual space SV based on the information on the positional relationship in the real space SR, and acquires user position information indicating the position of the user UV in the virtual space SV based on the positional relationship in the virtual space SV. According to this configuration, the position of the real object OR with respect to the user U can be made to coincide with the position of the virtual object OV with respect to the user UV. Therefore, the behavior of the user U in the virtual space SV can be made natural.

[0050] The display device according to the second aspect includes a providing unit 46 that provides the virtual space SV to the user U based on the user position information, and the providing unit 46 displays an image showing the real object OR within the virtual space SV so as to overlap the virtual object OV. Thereby, even when the sizes of the real object OR and the virtual object OV are different, the user U can recognize the range of the real object OR within the virtual space SV, so that the behavior of the user U in the virtual space SV can be made more natural.

[0051] In the display device according to the third aspect, the virtual object information acquisition unit 42 extracts the virtual object OV based on the shape of the real object OR and the shape of the virtual object OV. Thereby, by using the virtual object OV similar to the real object OR as a reference, the position of the user UV can be set more appropriately, so that the behavior of the user U in the virtual space SV can be made more natural.

[0052] In the display device according to the fourth aspect, the user position acquisition unit 44 acquires user position information when the user U performs an operation that satisfies a predetermined condition. As a result, the position of the user UV in the virtual space SV can be appropriately set as needed, so that the behavior of the user U in the virtual space SV can be made more natural.

[0053] The control method of the display device according to the fifth aspect includes steps of: acquiring information on the position of the real object OR, which is an object in the real space SR where the user U exists, in the real space SR, and information on the positional relationship between the real object OR and the user U in the real space SR; acquiring information on the position of the virtual object OV, which is an object in the virtual space SV, in the virtual space SV; aligning the positions and postures of the real object OR and the virtual object OV, calculating the positional relationship between the virtual object OV and the user UV in the virtual space SV based on the information on the positional relationship in the real space SR, and acquiring user position information indicating the position of the user UV in the virtual space SV based on the positional relationship in the virtual space SV.

[0054] The program according to the sixth aspect causes a computer to execute steps of: acquiring information on the position of the real object OR, which is an object in the real space SR where the user U exists, in the real space SR, and information on the positional relationship between the real object OR and the user U in the real space SR; acquiring information on the position of the virtual object OV, which is an object in the virtual space SV, in the virtual space SV; aligning the positions and postures of the real object OR and the virtual object OV, calculating the positional relationship between the virtual object OV and the user UV in the virtual space SV based on the information on the positional relationship in the real space SR, and acquiring user position information indicating the position of the user UV in the virtual space SV based on the positional relationship in the virtual space SV.

[0055] Although the embodiments of the present invention have been described above, the embodiments are not limited by the contents of these embodiments. Further, the above-described components include those that can be easily assumed by those skilled in the art, those that are substantially the same, and those within a so-called equivalent range. Furthermore, the above-described components can be combined as appropriate, and the configurations of each embodiment can also be combined. Furthermore, various omissions, substitutions, or changes of the components can be made without departing from the gist of the above-described embodiments.

Explanation of Signs

[0056] 1 Display system 10 Display device 12 Information processing device 20 Input unit 22 Display unit 24 Storage unit 26 Communication unit 28 Real space detection unit 30 Control unit 40 Real object information acquisition unit 42 Virtual object information acquisition unit 44 User position acquisition unit 46 Provision unit LR Real local space LV Virtual local space OR Real object OV Virtual object SR Real space SV Virtual space U,UV User V Field of view

Claims

1. An actual object information acquisition unit that acquires information on the position of an actual object, which is an object in the actual space where the user exists, and information on the positional relationship between the actual object and the user in the actual space; A virtual object information acquisition unit that acquires information on the position of a virtual object, which is an object in the virtual space, in the virtual space; A user position acquisition unit that aligns the positions and postures of the actual object and the virtual object, calculates the positional relationship between the virtual object and the user in the virtual space based on the information on the positional relationship in the actual space, and acquires user position information indicating the position of the user in the virtual space based on the positional relationship in the virtual space; including A display device.

2. including a providing unit that provides a virtual space to the user based on the user position information, The providing unit displays an image showing the actual object in the virtual space so as to overlap the virtual object. The display device according to claim 1.

3. The virtual object information acquisition unit extracts the virtual object based on the shape of the actual object and the shape of the virtual object. The display device according to claim 1 or claim 2.

4. The user position acquisition unit acquires the user position information when the user performs an operation that satisfies a predetermined condition. The display device according to any one of claims 1 to 3.

5. A step of acquiring information on the position of an actual object, which is an object in the actual space where the user exists, and information on the positional relationship between the actual object and the user in the actual space; A step of obtaining information on the position of a virtual object, which is an object in a virtual space, in the virtual space; Aligning the positions and postures of the real object and the virtual object, calculating the positional relationship in the virtual space between the virtual object and the user in the virtual space based on the information on the positional relationship in the real space, and obtaining user position information indicating the position of the user in the virtual space based on the positional relationship in the virtual space; Including A control method for a display device.

6. A step of obtaining information on the position of a real object, which is an object in the real space where the user exists, in the real space, and information on the positional relationship between the real object and the user in the real space; A step of obtaining information on the position of a virtual object, which is an object in the virtual space, in the virtual space; Aligning the positions and postures of the real object and the virtual object, calculating the positional relationship in the virtual space between the virtual object and the user in the virtual space based on the information on the positional relationship in the real space, and obtaining user position information indicating the position of the user in the virtual space based on the positional relationship in the virtual space, and causing a computer to execute; A program.

Images