Control device, system, control method, and program

Abstract

To make it possible to grasp the possibility that a virtual object is arranged at an inappropriate position when the virtual object is arranged according to a position of a user.SOLUTION: A control device for controlling a display device worn by a first user in a first real space includes control means for controlling the display device so as to display a virtual object as if the object has been arranged at a position in the first real space that corresponds to a position of a second user in a second real space, and a range display object indicating a range in which the virtual object can move in the first real space.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a control device, a system, a control method, and a program. [Background technology]

[0002] The same virtual object (AR object) may be displayed on the head mounted displays (HMDs) of multiple users located in different real spaces. Also, a virtual person (avatar) positioned according to the user's position in a second real space may be displayed on an HMD worn by a user located in a first real space. In this case, if the size of the first real space differs from the size of the second real space, restrictions arise on the position of the avatar in the HMD.

[0003] Patent Document 1 discloses a technique for, when a virtual object is placed in a position where it should not be placed, notifying a user that the virtual object has been placed in a position where it should not be placed. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2018-106298 A Summary of the Invention [Problem to be solved by the invention]

[0005] In this way, in Patent Document 1, after a virtual object is placed in a position where it should not be placed, the user is notified that the virtual object has been placed in a position where it should not be placed. For this reason, the user cannot know in advance that there is a possibility that the virtual object will be placed in a position where it should not be placed.

[0006] Therefore, an object of the present invention is to provide a technique that makes it possible to grasp the possibility that a virtual object will be placed in an inappropriate position when placing the virtual object according to the user's position. [Means for solving the problem]

[0007] One aspect of the present invention is a method for producing a composition comprising the steps of: A control device that controls a display device worn by a first user in a first real space, a control means for controlling the display device to display a virtual object that appears to be arranged at a position in the first real space corresponding to a position of a second user in a second real space, and a range display object that indicates a range within which the virtual object can be moved in the first real space; The control device is characterized by the above.

[0008] One aspect of the present invention is a method for producing a composition comprising the steps of: A control method for controlling a display device worn by a first user in a first real space, comprising: a control step of controlling the display device to display a virtual object that appears to be arranged at a position in the first real space corresponding to a position of a second user in a second real space, and a range display object that indicates a range within which the virtual object can be moved in the first real space, The control method is characterized by the above. Effect of the Invention

[0009] According to the present invention, when a virtual object is placed according to a user's position, it is possible to make it possible to grasp the possibility that the virtual object will be placed in an inappropriate position. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a diagram illustrating a system according to a first embodiment. [Figure 2A] FIG. 2 is a diagram for explaining an image of how the system according to the first embodiment is used; [Figure 2B] FIG. 2 is a diagram showing a display example of an HMD according to the first embodiment. [Figure 2C] FIG. 2 is a diagram showing a composite image generated by the camera according to the first embodiment. [Figure 3A] FIG. 2 is a diagram showing an example of a real space according to the first embodiment. [Figure 3B] FIG. 1 is a diagram for explaining a problem associated with the first embodiment. [Figure 4] FIG. 2 is a diagram for explaining a movable range of an avatar in the first embodiment. [Diagram 5] FIG. 1 is a configuration diagram of a video see-through HMD according to a first embodiment. [Figure 6] FIG. 1 is a configuration diagram of an optical see-through type HMD according to a first embodiment. [Figure 7] FIG. 1 is a configuration diagram of a camera according to a first embodiment. [Figure 8] FIG. 2 is a diagram for explaining a user position according to the first embodiment. [Figure 9] 5 is a flowchart showing a range information detection process according to the first embodiment. [Figure 10] FIG. 4 is a diagram showing a list related to obstacle objects according to the first embodiment. [Figure 11] FIG. 2 is a diagram for explaining an effective range according to the first embodiment. [Figure 12] 4A to 4C are diagrams illustrating gradation display of an effective range according to the first embodiment. [Figure 13] FIG. 4 is a diagram for explaining range information according to the first embodiment. [Figure 14] FIG. 13 is a diagram illustrating a system according to a second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Preferred embodiments of the present invention will now be described with reference to the accompanying drawings.

[0012] First, we will explain the technology related to HMD (head mounted display). HMD, smartphones, tablet terminals, etc. use technologies such as AR or MR. Here, an HMD that can be worn on the user's head has a display placed in front of the user's eyes. Therefore, the HMD can display useful information according to the usage scene and give the user a deep sense of immersion.

[0013] There are two types of HMDs: optical see-through HMDs that use a transparent (semi-transparent) display, and video see-through HMDs that use an opaque display.

[0014] The optical see-through type HMD allows the user to simultaneously view both the image and the incident light from the outside world. In other words, the optical see-through type HMD allows the user to view the outside space through the display. With the optical see-through type HMD, for example, the user can experience an event (such as a concert or an athletic meet) through the HMD at a certain location, while obtaining various information about people or objects that the user is paying attention to from the display on the HMD.

[0015] On the other hand, a video see-through HMD can display a virtual space on a display in front of the user's eyes, or can display an image captured by a camera mounted on the HMD on a display. This allows the HMD to display, for example, various information superimposed on an image captured of the real space in which the user is located.

[0016] <Embodiment 1> Fig. 1 is a diagram illustrating a system 1 (display system; control system) according to embodiment 1. In Fig. 1, a real space 101 (101A, 101B) is an actual space (real space) in which a user 100 (100A, 100B) is located.

[0017] System 1 has HMD 102A and camera 103B in real space 101A. System 1 has HMD 102B and camera 103B in real space 101B. HMD 102A and HMD 102B have the same configuration, and camera 103A and camera 103B have the same configuration. For this reason, in the following, after the description of one of HMD 102A and HMD 102B, the other will not be described as a general rule. Similarly, after the description of one of camera 103A and camera 103B, the other will not be described as a general rule.

[0018] The HMD 102A is an HMD worn by a user 100A (a user located in a real space 101A). Unless otherwise specified, the HMD 102A will be described on the assumption that it is a video see-through type HMD.

[0019] Camera 103A is an imaging device installed at a fixed position in real space 101A. Camera 103A captures real space 101A and user 100A in real space 101A. Camera 103A detects the position, posture, and facial expression (such as a smiling expression, an angry expression, or a neutral expression) of user 100A from an image (captured image) captured of user 100A (real space 101A). Camera 103A then transmits information on the detection result (the position, posture, and facial expression of user 100A) to server 107 as state information 10A.

[0020] At this time, the status information 10A is transmitted to the HMD 102B via the server 107. The HMD 102B then controls the avatar (the position, posture, facial expression, etc. of the avatar) displayed on the display (display unit) according to the received status information 10A. This allows the user 100B wearing the HMD 102B to recognize changes in the position, posture, and facial expression of the user 100A (the other user) in real time.

[0021] Furthermore, the camera 103A detects range information 20A indicating a range (effective range) in the real space 101A in which the user 100A can move and in which the user 100A can be detected from the captured image. The camera 103A then transmits the range information 20A to the HMD 102B via the server 107. In this case, the HMD 102B displays the range in which the avatar can move on the display based on the received range information 20A. Details of the state information 10 (10A, 10B) and the range information 20 (20A, 20B) will be described later.

[0022] (About the system usage image) An image of how the system 1 is used will be described with reference to FIGS. 2A to 2C.

[0023] FIG. 2A shows a situation in which two users 100 (100A, 100B) are using HMDs 102 (102A, 102B) in different real spaces 101 (a room 201 which is the real space 101A, and a garden 202 which is the real space 101B).

[0024] Camera 103A is placed in a corner of room 201. Camera 103B is placed in a corner of garden 202. Furthermore, a television 209 and a potted plant 210 are placed in room 201. In garden 202, a tree 211 and a dog 212 are placed.

[0025] Here, when the state (position, posture, facial expression, etc.) of the user 100A is reflected in the avatar of the user 100A, the range in which the user 100A can move within the room 201 (the range in which the user 100A can move within the room 201) is determined. The effective range of user 100A will be described. The effective range of user 100A is the range of room 201 excluding the range in which television 209 and potted plant 210 are placed, and is the range in which camera 103A can detect user 100A. The effective range of user 100B is the range of garden 202 excluding the range in which garden tree 211 and dog 212 are placed, and is the range in which camera 103B can detect user 100B.

[0026] 2B shows an example of an image displayed on the HMD 102 (102A, 102B) of the user 100 (100A, 100B). The HMD 102A displays an avatar 220B of the user 100B with the room 201 as the background. The HMD 102B displays an avatar 220A of the user 100A with the garden 202 as the background. The position, posture, expression, etc. of the avatar displayed on the HMD 102 change in response to changes in the state of other users (users other than the user wearing the HMD 102).

[0027] Here, in the image displayed on HMD 102A, avatar 220B is arranged so that the relative position of user 100B with respect to camera 103B coincides with the relative position of avatar 220B with respect to camera 103A. Similarly, in the image displayed on HMD 102B, avatar 220A is arranged so that the relative position of user 100A with respect to camera 103A coincides with the relative position of avatar 220A with respect to camera 103B.

[0028] FIG. 2C shows an example of an image in which an avatar is synthesized into an image (captured image) captured by camera 103 (camera 103A, camera 103B) of real space 101. Here, camera 103 (103A, camera 103B) captures only the user and the background. Camera 103 generates a composite image in which the user and the avatar appear to be in the same real space by synthesizing the captured image with an avatar (an avatar generated based on state information of other users). The composite image generated in this manner is recorded in server 107 or distributed to an external device via server 107. This allows the user and a third party other than the user to view a composite image in which the user and the avatar appear to be playing together. In the following, the composite image generated by camera 103 is referred to as a "camera composite image."

[0029] Furthermore, in the camera composite image generated by camera 103A, avatar 220B is positioned so that the relative position of user 100B with respect to camera 103B coincides with the relative position of avatar 220B with respect to camera 103A. Similarly, in the camera composite image generated by camera 103B, avatar 220A is positioned so that the relative position of user 100A with respect to camera 103A coincides with the relative position of avatar 220A with respect to camera 103B.

[0030] For this reason, if the size of the real space 101 where each user is located is different, the image displayed on the HMD 102 and the camera composite image may look unnatural. For example, as shown in FIG. 3A, consider a case where the user 100B moves to a corner of the garden 202, which corresponds to the outside of the range of the room 201 (the effective range of the user 100A in the room 201). In this case, in the image displayed on the HMD 102A of the user 100A, the avatar 220B of the user 100B is located outside the room 201 of the user 100A, as shown in FIG. 3B. Also, in the camera composite image generated by the camera 103A, the avatar 220B of the user 100B is located outside the room 201, as shown in FIG. 3B.

[0031] Note that user 100B does not know the situation in user 100A's room 201. For this reason, user 100B does not notice that the position of avatar 220B is unnatural in the image displayed on HMD 102A and the camera composite image generated by camera 103A. In other words, unless user 100B actually sees room 201 and understands the details of room 201, he or she cannot understand the range in which avatar 220B is not placed in an unnatural position (the range in which avatar 220B is allowed to move).

[0032] To solve this problem, in the first embodiment, an HMD 102 that displays a range in which the avatar 220B can move as a range in which the movement of the user 100A is permitted, based on range information 20B in the real space 101B, will be described.

[0033] 4A and 4B show examples of the range in which an avatar displayed on the HMD 102 can move. In FIG. 4A, a virtual object (hereinafter referred to as a "range display object") 401 showing the range in which the avatar 220B of the user 100B can move by gradation display (shading display) is displayed in the HMD 102A. In FIG. 4B, a range display object 403 showing the range in which the avatar 220A of the user 100A can move by gradation display is displayed in the HMD 102B. As will be described later, the darker the range display object 403 is, the more accurately the state of the user 100A is detected in the range of the real space 101A corresponding to the darker range, so that the avatar 220A moves smoothly. In addition, the range display object 403 has transparency, and the user 100B can see the real space 101B (an image of the real space 101B) through the range display object 403.

[0034] 4A from the display of HMD 102A, and acts within the range of real space 101A displayed by gradation. This makes it possible to prevent avatar 220A from being placed at an unnatural position in the image displayed on HMD 102B and the composite image synthesized by camera 103B. Specifically, since the range of range display object 401 corresponds to the effective range of user 100B, if user 100A moves within the range of range display object 401, avatar 220A moves only within the effective range of user 100B.

[0035] (About the HMD configuration) Next, the internal configuration of the HMD 102 (102A, 102B) will be described. In the above, it has been assumed that the HMD 102 is a video see-through type HMD. However, the HMD 102 may be either an optical see-through type HMD or a video see-through type HMD. Here, each component of the HMD 102 is controlled by a control unit (not shown). In other words, the control unit controls the entire HMD 102 (display device).

[0036] First, a case where the HMD 102 is a video see-through type HMD 500 will be described with reference to Fig. 5. The HMD 500 has an imaging unit 501, an acquisition unit 502, an object generation unit 503, a superimposition unit 504, and a display unit 505. In the following description, it is assumed that the HMD 500 is the HMD 102A worn by the user 100A.

[0037] The imaging unit 501 is an imaging device (camera) that acquires an image (hereinafter referred to as a “front image”) of the front of the user 100A. The imaging unit 501 generally uses an imaging device having an imaging angle of view (ranging from a wide angle to a standard imaging angle of view) close to the field of view of the user 100A.

[0038] The acquisition unit 502 acquires the state information 10B and the range information 20B from the camera 103B of the user 100B via the server 107. The acquisition unit 502 transmits the state information 10B and the range information 20B to the object generation unit 503.

[0039] The position of user 100B indicated by state information 10B is the relative position of user 100B with respect to camera 103B. Range information 20B is information indicating the effective range of user 100B (the range in which user 100B can move in real space 101B and the range in which camera 103B can detect user 100B) by the relative position from camera 103B. Range information 20B also includes information on the gradation level (density) of each position in the effective range according to the detection accuracy of the state of user 100B at that position.

[0040] The object generating unit 503 generates avatar 220B (an image of avatar 220B) based on state information 10B received from acquisition unit 502. Furthermore, the object generating unit 503 generates a range display object (an image of the range display object) indicating a range in which avatar 220B can move based on range information 20B received from acquisition unit 502. Specifically, the object generating unit 503 generates a range display object that can cover a range of the forward image (real space 101A) that corresponds to the effective range indicated by range information 20B. Furthermore, the object generating unit 503 colors each position of the range display object according to the gradation level of the position of the effective range that corresponds to that position.

[0041] Superimposing unit 504 generates a composite image (see FIGS. 4A and 4B) by superimposing avatar 220B (image of avatar 220B) and a range indicating object (image of range indicating object) on an image (front image) captured by imaging unit 501. Then, superimposing unit 504 outputs the composite image to display unit 505.

[0042] At this time, the superimposing unit 504 places the avatar 220B at a position (position in the forward image) according to the position of the user 100B indicated by the state information 10B. Specifically, the superimposing unit 504 places the avatar 220B so that the relative position of the user 100B with respect to the camera 103B coincides with the relative position of the avatar 220B with respect to the camera 103A. For this reason, for example, the superimposing unit 504 acquires in advance the relative position of the camera 103A with respect to the HMD 500, and places the avatar 220B based on the relative position and the state information 10B.

[0043] The display unit 505 is a display provided in front of the user's eyes and displays the composite image.

[0044] 6, a case where the HMD 102 is an optical see-through HMD 600 will be described. The HMD 600 has an acquisition unit 601, an object generation unit 602, and a projection unit 603. In the following, it is assumed that the HMD 600 is the HMD 102A worn by the user 100A.

[0045] In addition, in the optical see-through type HMD 600, the user can directly view the real space 101A through a display surface (display; glasses). For this reason, the HMD 600 does not include the imaging unit 501.

[0046] The acquisition unit 601 acquires, via the server 107, the status information 10B and the range information 20B from the camera 103B of the user 100B.

[0047] Similar to the video see-through HMD 500, the object generation unit 602 generates the avatar 220B (an image of the avatar 220B) and a range display object.

[0048] The projection unit 603 projects the avatar 220B and the range display object onto an optical element (such as a prism) installed in the display. At this time, the projection unit 603 projects (places) the avatar 220B at a position (position on the display) according to the position of the user 100B indicated by the state information 10B. This allows the user to project (place) the avatar 220B in the real space 101A. The user can see (recognize) the space in which the avatar 220B and the range display object are located.

[0049] As long as the HMD 102 includes the configuration shown in FIG. 5 or FIG. 6, the shape of the HMD 102 may be any shape, such as a goggle shape, a eyeglass shape, or a contact lens shape.

[0050] (Camera configuration) The internal configuration of cameras 103 (103A, 103B) will be described with reference to Fig. 7. Camera 103A that captures an image of user 100A will be described below, but camera 103B has the same configuration as camera 103A.

[0051] The imaging section 701 images the user 100A in the real space 101A. The imaging section 701 is capable of imaging, for example, a wide range of the real space 101A.

[0052] The detection unit 702 detects the user 100A in the real space 101A based on an image (captured image) of the user 100A captured by the imaging unit 701. Then, the detection unit 702 acquires information (status information 10A and range information 20A) of the user 100A based on the captured image. Note that the detection unit 702 may acquire the range information 20A acquired the previous time the camera 103A was used, for example, from the recording unit 707, instead of acquiring the range information 20A based on the captured image.

[0053] The state information 10A is information about the state of the user 100A (position, posture, face direction, facial expression, etc.). The position of the user 100A is represented by the coordinate position where the user 100A is detected in a coordinate space having the position of the camera 103A placed in the real space 101A as the origin (0,0,0) as shown in FIG. 8. The posture of the user 100A is estimated using a technique such as Deep Learning based on the coordinate positions of the four limbs of the user 100A in the coordinate space shown in FIG. 8. The face direction is detected based on whether the face is facing up, down, left, or right, assuming that a state in which the face of the user 100A faces the camera 103A directly is a "front facing state". The facial expression is estimated from the detection results of the degree to which the user 100A's eyes are open and the position of the corners of the mouth. While the imaging section 701 is imaging the user 100A, the detection section 702 detects the position, posture, facial direction and facial expression of the user 100A at a constant rate (period), and updates the state information 10A.

[0054] Range information 20A is information indicating the effective range of user 100A (the range in real space 101A in which user 100A can move and in which user 100A can be detected by camera 103A). A method for detecting range information 20A will be described later with reference to the flowchart of FIG.

[0055] The transmission unit 703 transmits the state information 10A and the range information 20A to the server 107. The transmission unit 703 is a communication device. The transmission unit 703 also transmits the camera composite image generated by the superimposing unit 706 to the server 107.

[0056] The acquiring unit 704 acquires the status information 10B of the user 100B via the server 107. The acquiring unit 704 is a communication device.

[0057] The object generating unit 705 generates avatar 220B (an image of avatar 220B) whose position, posture, facial expression, and the like are controlled based on the state information 10B.

[0058] The superimposing unit 706 superimposes the avatar 22 on the captured image of the user 100A captured by the imaging unit 701. 0B to generate a camera composite image. At this time, superimposition unit 706 places avatar 220B at a position (position in the captured image) according to the position of user 100B indicated in state information 10B. Specifically, avatar 220B is placed such that the relative position of user 100B with respect to camera 103B and the relative position of avatar 220B with respect to camera 103A match in the camera composite image.

[0059] The recording unit 707 stores the camera composite image (an image in which the avatar 220B is superimposed on the captured image). The recording unit 707 may also store the state information 10A and the range information 20A acquired by the detection unit 702.

[0060] (Regarding range information detection process) 9, a process for detecting range information 20 executed by detection unit 702 will be described. In the following, a process executed by detection unit 702 of camera 103A that captures an image of user 100A (real space 101A) will be described.

[0061] In step S901, the detection unit 702 detects an object (obstacle object) that impedes the movement of the user 100A from an image (captured image) of the user 100A captured by the imaging unit 701. In Fig. 2A, the obstacle objects correspond to the television 209 and the potted plant 210 in the room 201. Also, the obstacle objects correspond to the garden tree 211 and the dog 212 in the garden 202.

[0062] Here, the detection unit 702 sets a three-dimensional coordinate space with the position of the camera 103A in the real space 101A as the origin (0,0,0) as shown in Fig. 8. Then, the detection unit 702 detects the position and size (width W and height H) of an obstacle object in the coordinate space. Note that, in order to improve the detection accuracy of the obstacle object, general recognition technology such as AI (Artificial Intelligence) and DL (Deep Learning) may be used.

[0063] Then, based on the detection result of the detected objects, the detection unit 702 generates a list indicating the position and size (the vertical and horizontal lengths of the obstacle objects when viewed from the Z-axis direction) of each obstacle object as shown in Fig. 10. Note that the information on the position and size of the obstacle objects may be registered in advance by the user.

[0064] In step S902, the detection unit 702 detects a range (effective range) in the real space 101A in which the user 100A can move (can move without being hindered by obstacle objects) and in which the user 100A can be detected (imaged) by the camera 103A.

[0065] First, as shown in FIG. 11, the detection unit 702 sets a two-dimensional coordinate space (a two-dimensional coordinate space of the real space 101A as viewed from the Z-axis direction in FIG. 8) with the position of the camera 103A in the real space 101A as the origin (0,0). Then, the detection unit 702 obtains a range on the camera 103A side from a boundary line 1100 (a boundary line of a range in which the user 100A can move; such as a wall) of the real space 101A, among the ranges included in the imaging angle of view of the camera 103A in the set coordinate space. Then, the detection unit 702 detects a range obtained by removing the object 1103 and the blind spot range 1104 (a range that cannot be seen by the camera 103A due to the presence of the object 1103) from the obtained range as an effective range 1105 (a range indicated by diagonal lines). For this reason, the effective range 1105 does not include the left and right blind spot ranges 1101 and 1102 that are not included in the imaging angle of view of the camera 103A. The blind spot range 1104 can be calculated by a known method from the position and size of each obstacle object shown in FIG. 10 (that is, the position and size of the object 1103).

[0066] In step S903, detection unit 702 adds information about the detection accuracy of the state of user 100A (position, posture, facial expression, etc. of user 100A) within the imaging angle of view of camera 103A to the valid range detected in step S902.

[0067] As shown in FIG. 12A, when the effective range 1105 is detected, the user 100A can freely move inside the effective range 1105. However, in order for the camera 103A to accurately detect the state of the user 100A (position, posture, facial expression, etc.), it is necessary to capture the entire body of the user 100A at an appropriate size. For example, when capturing an image of the user 100A located in the range 1201 shown in FIG. 12A, the user 100A is too close to the camera 103A, so only a part of the body of the user 100A can be captured. This reduces the detection accuracy of the state of the user 100A by the detection unit 702. Also, when capturing an image of the user 100A located in the range 1202, the user 100A is too far from the camera 103A, so the whole body of the user 100A is captured small. This reduces the detection accuracy of the state of the user 100A in this case as well.

[0068] In this way, the detection accuracy of the state of the user 100A is a value based on the size of the user 100A in the captured image and / or the range of the whole body of the user 100A that is captured in the captured image. Therefore, the detection accuracy of the actual state of the user 100A gradually changes according to the change in the distance between the position of the user 100A and the camera 103A. FIG. 12B shows the detection accuracy of the state of the user 100A corresponding to each coordinate of the effective range 1105 in FIG. 12A by displaying it in gradation. The dark range of the gradation is the range in which the user 100A can move and the detection accuracy of the state of the user 100A by the camera 103A is also high. The light range of the gradation is the range in which the user 100A can move but the detection accuracy of the state of the user 100A by the camera 103A is low. Therefore, if the user 100A is located in the light range, the movement of the avatar 220A displayed on the HMD 102B may stop or the avatar 220A may not be in the correct position or posture.

[0069] Therefore, detection unit 702 adds information related to the detection accuracy of user 100A's state to the effective range detected in step S902, and outputs the result as range information 20A. Specifically, detection unit 702 outputs information expressing the detection accuracy of user 100A's state at each coordinate position by a gradation level (density) as shown in Fig. 13, as range information 20A. Here, in Fig. 13, positions where the gradation level is greater than a specific value (e.g., 0) are included in the effective range, and positions where the gradation level is equal to or less than the specific value are not included in the effective range.

[0070] As in steps S901 to S903, the detection unit 702 detects the effective range of the user 100A in the real space 101A from the captured image, and transmits range information 20A to the HMD 102B and camera 103B of the user 100B. Then, based on the range information 20A, the HMD 102B sets a coordinate space with the position of the camera 103B placed in the real space 101B as the origin, as shown in FIG. 8. The HMD 102B generates a range display object that colors the coordinate positions (coordinate positions in the coordinate space of the real space 101B) corresponding to each position of the effective range according to the gradation level of the corresponding position of the effective range (the higher the gradation level, the darker the color). Then, the HMD 102B displays the range display object together with the avatar 220A.

[0071] Therefore, each position of the range display object is displayed in a display form according to the detection accuracy of the state of the user 100A at the coordinate position in the real space 101A corresponding to that position. Note that each position of the range display object does not need to be displayed by gradation display, and may be displayed in a different color according to the detection accuracy, or may be displayed in a pattern according to the detection accuracy, for example.

[0072] In this embodiment, the detection process of range information 20 representing a two-dimensional coordinate space has been described, but range information 20 representing a three-dimensional coordinate space taking into account the height direction of real space may be detected. Moreover, acquisition of range information 20 is usually performed only once at the timing when display of avatar 220 starts.

[0073] As described above, according to the first embodiment, the user 100A can recognize the movement range of the avatar 220B and the detection accuracy of the state of the user 100B by looking at the information (range display object) indicating the movement range of the avatar 220B of the other user 100B. Therefore, the user 100A can grasp the possibility that the avatar 220B will be placed in an inappropriate position. Then, the user 100A can avoid the avatar 220A of the user 100A being placed in an unnatural position in the image seen by the user 100B by acting within the range indicated by the range display object. Furthermore, the user 100A can avoid the avatar 220A of the user 100A from making an unnatural movement in the image seen by the user 100B by acting within the dark range of the gradation display of the range display object.

[0074] <Embodiment 2> In the second embodiment, the HMD 102 also displays a movable range of the user 100 himself / herself who is wearing the HMD 102. In the following description, the HMD 102 will be described as an HMD 102A worn by a user 100A.

[0075] In the first embodiment, the movable range of the avatar 220B of the user 100B is displayed on the HMD 102A. Here, when the video see-through type HMD 102A displays an image of a virtual space (a space that is not a real space), it is possible that the user 100A plays with the avatar 220B in the virtual space, or that the scene of them playing is captured. When an image of the virtual space is displayed on the HMD 102A, the user 100A cannot visually recognize the real space 101A in which the user 100A is currently located with the naked eye. Therefore, when the user 100A plays a game involving movement using the HMD 102A, there is a risk that the user 100A will collide with an obstacle object arranged in the real space 101A.

[0076] To avoid this, the HMD 102A displays not only the moving range of the avatar 220B of the user 100B but also the movable range of the user 100A based on the range information 20A detected by the camera 103A by gradation display (superimposing it on the image of the virtual space). That is, the HMD 102A displays a range display object indicating a range (range in the image of the virtual space) corresponding to the movable range of the user 100A himself in the real space 101A. Note that the "movable range of the user 100A" in the second embodiment may be the same range as the effective range of the user 100A. The "movable range of the user 100A" may be a range that combines the effective area of ​​the user 100A and the range that cannot be seen from the camera 103A due to the presence of an obstacle object (blind spot range 1104 in FIG. 11).

[0077] 14 shows the configuration of a system 2 in embodiment 2. The basic configuration of a system 1 is similar to that of the system 2 according to embodiment 1. Meanwhile, the HMD 102A also acquires range information 20A of a user 100A wearing the HMD 102A in order to know the movable range of the user 100A himself / herself.

[0078] The HMD 102A according to the second embodiment is similar to the see-through HMD 500 (FIG. 5), but the object generation unit 503 generates range display objects representing movable ranges for each of the two pieces of range information 20A and 20B. The HMD 102A may individually switch between the two range display objects at regular time intervals, or may display different range display objects. It may also be displayed simultaneously using a color gradation.

[0079] This allows user 100A to recognize the movable range of both himself / herself and avatar 220B, and therefore allows user 100A to avoid danger such as colliding with an obstacle object.

[0080] In the second embodiment, an example is given in which both users use a video see-through HMD to display their avatars in a virtual space, but only one of the users' HMDs may be a video see-through HMD.

[0081] In each of the above embodiments, the HMD (display device) may be composed of a control device that controls the HMD (for example, a configuration in which the display unit 505 is removed from the HMD 500) and a display unit (for example, the display unit 505 in the HMD 500).

[0082] Also, in the above, "If A is equal to or greater than B, proceed to step S1, and if A is smaller (lower) than B, proceed to step S2" may be read as "If A is greater (higher) than B, proceed to step S1, and if A is equal to or less than B, proceed to step S2." Conversely, "If A is greater (higher) than B, proceed to step S1, and if A is equal to or less than B, proceed to step S2" may be read as "If A is greater (higher) than B, proceed to step S1, and if A is smaller (lower) than B, proceed to step S2." For this reason, unless a contradiction occurs, the expression "equal to or greater than A" may be read as "A or greater (high; long; many)," or may be read as "greater than A (high; long; many)." On the other hand, the expression "equal to or less than A" may be read as "A or smaller (low; short; few)," or may be read as "smaller than A (low; short; few)." Furthermore, "bigger (higher; longer; more) than A" may be read as "A or greater," and "smaller (lower; shorter; fewer) than A" may be read as "A or less."

[0083] Although the present invention has been described in detail based on the preferred embodiments, the present invention is not limited to these specific embodiments, and various forms within the scope of the gist of the present invention are also included in the present invention. Parts of the above-described embodiments may be combined as appropriate.

[0084] Each functional unit in each of the above embodiments (variations) may or may not be individual hardware. The functions of two or more functional units may be realized by common hardware. Each of a plurality of functions of one functional unit may be realized by individual hardware. Two or more functions of one functional unit may be realized by common hardware. Furthermore, each functional unit may or may not be realized by hardware such as an ASIC, FPGA, or DSP. For example, the device may have a processor and a memory (storage medium) in which a control program is stored. Then, the functions of at least some of the functional units of the device may be realized by the processor reading and executing the control program from the memory.

[0085] (Other embodiments) The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) for implementing one or more of the functions.

[0086] The disclosure of the above embodiments includes the following configurations, methods, and programs. [Configuration 1] A control device that controls a display device worn by a first user in a first real space, a control means for controlling the display device to display a virtual object that appears to be arranged at a position in the first real space corresponding to a position of a second user in a second real space, and a range display object that indicates a range within which the virtual object can be moved in the first real space; A control device comprising: [Configuration 2] a range in which the virtual object can be moved in the first real space is a range based on a captured image acquired by an imaging device in the second real space; 2. The control device according to configuration 1, [Configuration 3] a range in which the virtual object can be moved in the first real space corresponds to an effective range; the effective range is a range in which the second user can move in the second real space and a range in which the second user can be detected from the captured image; 3. The control device according to configuration 2. [Configuration 4] the control means displays, in the range display object, each position within a range in which the virtual object can be moved in the first real space in a display form corresponding to a detection accuracy of a state of the second user at a position in the second real space corresponding to each position. 4. The control device according to configuration 3. [Configuration 5] The detection accuracy of the state of the second user is based on at least one of a range of the whole body of the second user that appears in the captured image and a size of the second user in the captured image. 5. The control device according to configuration 4. [Configuration 6] An acquisition means for acquiring range information indicating the effective range; a generating means for generating the range display object based on the range information; 6. The control device according to any one of configurations 3 to 5, further comprising: [Configuration 7] The acquiring means further acquires status information indicating a status of the second user including a location of the second user, The generating means generates the virtual object based on the state information. having 7. The control device according to configuration 6, [Configuration 8] The display device is a display device that allows the outside to be seen through a display, the control means controls the display device so as to display the virtual object and the range display object on the display. 8. The control device according to any one of configurations 1 to 7. [Configuration 9] the control means controls the display device to display an image in which the virtual object and the range indication object are combined with an image capturing an area in front of the first user. 8. The control device according to any one of configurations 1 to 7. [Configuration 10] The control means displays an image of a virtual space, and displays a range in the image of the virtual space that corresponds to a range in which the first user can move in the first real space. controlling the display device to further display a third image indicative of a range. 10. The control device according to any one of configurations 1 to 9. [Configuration 11] A control device according to any one of configurations 1 to 10; an imaging device that captures an image of the second real space and obtains a captured image; A system comprising: [Configuration 12] a detection means for detecting an effective range in the second real space based on the captured image, the effective range being a range in which the second user can move and a range in which the second user can be detected from the captured image; a range in which the virtual object can be moved in the first real space corresponds to the effective range; 12. The system according to claim 11, [Control method] A control method for controlling a display device worn by a first user in a first real space, comprising: a control step of controlling the display device to display a virtual object that appears to be arranged at a position in the first real space corresponding to a position of a second user in a second real space, and a range display object that indicates a range within which the virtual object can be moved in the first real space, A control method comprising: [program] A program for causing a computer to function as each of the means of the control device according to any one of configurations 1 to 10. [Explanation of symbols]

[0087] 1: System, 102: HMD (display device), 103: Camera (imaging device)

Claims

1. A control device for controlling a display device worn by a first user in a first real space, having control means for controlling the display device to display a virtual object arranged at a position in the first real space corresponding to the position of a second user in a second real space and a range display object indicating a range in which the virtual object can move in the first real space, the range in which the virtual object can move in the first real space corresponds to an effective range, the effective range is a range in which the second user can move in the second real space and is a range in which the second user can be detected from a captured image acquired by imaging by an imaging device in the second real space, A control device characterized by the above.

2. In the range display object, the control means displays each position in the range in which the virtual object can move in the first real space in a display form corresponding to the detection accuracy of the state of the second user at the position in the second real space corresponding to the position. The control device according to claim 1, characterized by the above.

3. The detection accuracy of the state of the second user is an accuracy based on at least either the range of the whole body of the second user shown in the captured image or the size of the second user in the captured image. The control device according to claim 2, characterized by the above.

4. An acquisition means for acquiring range information indicating the effective range, A generation means for generating the range display object based on the range information, The control device according to any one of claims 1 to 3, further characterized by the above.

5. The acquisition means further acquires state information indicating the state of the second user including the position of the second user, The generation means generates the virtual object based on the state information. The control device according to claim 4, characterized in that.

6. The display device is a display device that can be visually recognized from the outside through the display. The control means controls the display device to display the virtual object and the range display object on the display. The control device according to any one of claims 1 to 5, characterized in that.

7. The control means controls the display device to display an image obtained by synthesizing the virtual object and the range display object on an image captured in front of the first user. The control device according to any one of claims 1 to 6, characterized in that.

8. The control means controls the display device to display an image of the virtual space and further display a third image indicating a range in the image of the virtual space, which corresponds to the range within which the first user can move in the first real space. The control device according to any one of claims 1 to 7, characterized in that.

9. The control device according to any one of claims 1 to 8, An imaging device that captures the second real space and acquires a captured image, A system characterized by comprising.

10. Further comprising detection means for detecting the effective range based on the captured image. The system according to claim 9, characterized in that.

11. A control method for controlling a display device worn by a first user in a first real space, A control step of controlling the display device to display a virtual object arranged at a position in the first virtual space corresponding to the position of the second user in the second real space, and a range display object indicating a range in which the virtual object can move in the first real space. The range in which the virtual object can move in the first real space corresponds to a valid range. The valid range is a range in which the second user can move in the second real space and a range in which the second user can be detected from a captured image obtained by imaging with an imaging device in the second real space. A control method characterized by this.

12. A program for causing a computer to function as each means of the control device according to any one of claims 1 to 8.

Images