Information processing system

Abstract

To provide a technology capable of realizing both of easiness in grasping an instruction position of another user and preventing lowering of visibility in an operation on a three-dimensional space.SOLUTION: An information processing system includes: an image generation unit which generates a first image indicating a field view from a view point of a first user regarding a virtual three-dimensional space shared by a plurality of users, and combines the first image and an instruction UI used by each user to point a point in the three-dimensional space; a determination unit which determines visibility of the instruction point of a second user being a point in the three-dimensional space pointed by the second user with the instruction UI, where the visibility is obtained from the view point of the first user; and a display control unit which switches a display method of the instruction UI of the second user in the first image according to the determination result by the determination unit.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an information processing system. [Background technology]

[0002] Conventionally, there is a system that can connect computers via a LAN (Local Area Network) or the like to hold an online conference in which multiple people can participate. One of the functions of this online conference is a screen sharing function that allows multiple participants to share the same display screen and operate it simultaneously in real time. For example, Patent Document 1 proposes a technology in the screen sharing function that displays the position (indicated position) that each user is pointing to with a pointer, making it possible to visually easily distinguish the indicated positions of each user.

[0003] In recent years, space sharing systems have been introduced in which multiple users share a virtual three-dimensional space using technologies such as VR (Virtual Reality) and MR (Mixed Reality). Since three-dimensional space also includes depth information, a situation may occur in which the pointing position of one user is located behind an object (in a blind spot) when viewed from another user. Therefore, even if the technology of Patent Document 1 is applied directly to a three-dimensional space, the pointers of other users may not be visible, and it may be impossible to perceive where other users are pointing.

[0004] One way to solve this problem is to display a ray of light from the user's position (for example, the position of the hand or the position of the viewpoint) to the position pointed to by the user on the display screen in VR or MR. Even if the position pointed to by another user is hidden by an object, it is possible to roughly recognize where the other user is pointing by displaying the ray extending from the other user. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Publication No. 3-119478 Summary of the Invention [Problem to be solved by the invention]

[0006] However, because rays occupy a larger proportion of space than pointers, there is a problem that if multiple people are working together and all rays are displayed, the rays of other users may get in the way and reduce visibility.

[0007] In addition, VR and MR are sometimes used to display large objects such as cars and large machines as CG (Computer Graphics), and the designated position may be far from other users. In such cases, the pointer may be small and difficult for other users to see the designated position. However, if the pointer is made large so that it can be seen by users far away, there is an issue that the pointer gets in the way of people close to the designated position and makes it difficult to see the CG.

[0008] The present invention has been made to solve the above-mentioned problems, and aims to provide a technology that makes it possible to easily grasp the position of other users' indications while preventing a decrease in visibility, even when operating in three-dimensional space. [Means for solving the problem]

[0009] The present disclosure includes an information processing system having an image generation unit that generates a first image representing a field of view from a first user's viewpoint for a virtual three-dimensional space shared by a plurality of users and synthesizes an instruction UI (user interface) used by each user to point to a point in the three-dimensional space onto the first image; a determination unit that determines visibility of a second user's instruction position, which is a point in the three-dimensional space pointed to by a second user with the instruction UI, when viewed from the first user's viewpoint; and a UI display control unit that switches a display method of the second user's instruction UI in the first image depending on a determination result by the determination unit. Effect of the Invention

[0010] According to the present invention, it is possible to provide a technique that makes it possible to easily grasp the pointing position of other users while preventing a decrease in visibility in operations in a three-dimensional space. [Brief description of the drawings]

[0011] [Figure 1] FIG. 1 is a block diagram showing an example of the configuration of an information processing system according to a first embodiment; [Diagram 2] FIG. 1 is a diagram illustrating an operation device according to a first embodiment. [Diagram 3] FIG. 1 is a diagram for explaining a direction of instruction using an operation device according to a first embodiment; [Figure 4] 1 is a flowchart showing the operation of an information processing system according to a first embodiment. [Diagram 5] FIG. 1 is a diagram for explaining ray display according to the first embodiment; [Figure 6] FIG. 13 is a block diagram showing an example of the configuration of an information processing system according to a second embodiment. [Figure 7] Diagram showing a magnetic field sensor system [Figure 8] Block diagram showing the hardware configuration of the information processing system [Figure 9] 11 is a flowchart showing the operation of an information processing system according to a second embodiment. [Figure 10] A diagram showing how to determine the indicated position [Figure 11] A diagram showing an example of indicating position [Figure 12] FIG. 13 is a diagram showing a method for determining whether or not a pointed position is visible. [Figure 13] 11 is a flowchart showing the operation of an information processing system according to a second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] The present invention relates to a space sharing system in which multiple users share a virtual three-dimensional space, and more specifically, to a technology for improving a method of displaying the position and direction pointed by each user in a virtual three-dimensional space. The fusion of the real world and the virtual world is called cross reality (XR), and XR includes VR (virtual reality), AR (augmented reality), MR (mixed reality), SR (alternative reality), etc. The present invention is applicable to any type of XR content.

[0013] In the embodiment of the present invention described below, a user participates in a space sharing system by using an information processing system. The information processing system is a device that each user possesses and operates individually, and therefore may be called an information terminal, a user terminal, a client device, an edge, an XR terminal, or the like. Note that the space sharing system may be configured in a server-client manner in which each user's information processing system accesses a central server, or in a P2P manner in which each user's information processing system communicates with each other peer-to-peer, but any configuration may be used.

[0014] For example, assume that a first user and a second user share a virtual three-dimensional space. In a first information processing system operated by the first user, a first image representing the field of view from the first user's viewpoint is generated as an image to be shown to the first user. Similarly, in a second information processing system operated by a second user, a second image representing the field of view from the second user's viewpoint is generated as an image to be shown to the second user. In this case, even if the first user and the second user are looking at the same object O in the three-dimensional space, the viewpoints are different. Therefore, the first image and the second image will be different, and the object O will look different.

[0015] For example, assume a scene in which a first user and a second user are having a discussion while pointing at details of an object O. Each user uses an instruction UI (user interface) to point to a point in a three-dimensional space. The instruction UI may include, for example, a pointer representing a point (instruction position) in the three-dimensional space to which the user points, and a ray representing a direction (instruction direction) in which the user points. Here, the first information processing system synthesizes not only the instruction UI of the first user (oneself) but also the instruction UI of the second user (other person) into the first image, so that the first user can visually recognize the instruction position and instruction direction of the second user (other person). Similarly, the second information processing system synthesizes the instruction UI of the first user into the second image, so that the second user can visually recognize the instruction position and instruction direction of the first user. This makes it possible for each user to recognize the instruction position and instruction direction of the other users in the virtual three-dimensional space.

[0016] However, as mentioned in the previous issue, other users' rays and pointers may reduce the visibility of the object O, or may cause visual annoyance or obstruction. In addition, when users are far apart or point far away, it may be difficult to determine where the instruction UI is pointing. Such problems become more pronounced as the number of participating users increases.

[0017] Therefore, in the first information processing system, the visibility of the instruction position of the second user when viewed from the viewpoint of the first user is determined, and a UI display control is executed to switch the display method of the instruction UI of the second user in the first image according to the determination result. Similarly, in the second information processing system, the visibility of the instruction position of the first user when viewed from the viewpoint of the second user is determined, and a UI display control is executed to switch the display method of the instruction UI of the first user in the second image according to the determination result. In this way, by adaptively controlling the UI display according to the visibility, it is possible to achieve both ease of grasping the instruction position of other users and prevention of a decrease in visibility. A specific example of switching the display method of the instruction UI will be described in detail in the following embodiment.

[0018] Note that the operation method of the pointing direction (ray direction) and the pointing position (pointer position) does not matter. For example, the user's pointing direction and pointing position may be identified by detecting the position and orientation of an operation device worn or held by the user on his / her hand. Alternatively, the user's pointing direction and pointing position may be identified by recognizing the orientation and shape of the user's fingers using hand tracking technology using a camera. Alternatively, the user's pointing direction and pointing position may be identified by detecting the user's line of sight and gaze point. Furthermore, these multiple operation methods may be combined or switched depending on the situation.

[0019] [First embodiment] An information processing system according to a first embodiment will be described below with reference to the configuration diagram of Fig. 1. The information processing system 1 includes an HMD (Head Mounted Display) 100 and an operation device 120.

[0020] The HMD 100 is a head-mounted display device (electronic device) that can be worn on the head of a user. The HMD 100 includes an HMD control unit 101, an imaging unit 102, a position and orientation estimation unit 103, a depth map generation unit 104, a pointer position calculation unit 105, a UI display control unit 106, and a determination unit 112. The HMD 100 further includes a device communication unit 107, a server communication unit 108, an image generation unit 109, an image display unit 110, and a memory 111. The HMD control unit 101 controls each component of the HMD 100.

[0021] The imaging unit 102 may include two cameras (imaging devices). The two cameras are arranged near the left and right eyes of the user when wearing the HMD 100 in order to capture an image similar to that seen by the user's naked eye (when the HMD 100 is not worn). Images of a subject (a range in front of the user) captured by the two cameras are output to the image generation unit 109 and the position and orientation estimation unit 103. In the first embodiment, a configuration in which an image used by the image generation unit 109 and an image used by the position and orientation estimation unit 103 are shared will be described, but a plurality of cameras may be mounted, for example, by using different cameras for the image generation unit 109 and the position and orientation estimation unit 103.

[0022] The position and orientation estimation unit 103 receives images captured by the two cameras of the imaging unit 102, and estimates the position and orientation of the HMD 100 by Visual SLAM (Simultaneous Localization and Mapping). The estimated position and orientation information is transmitted to an image generation unit 109.

[0023] The depth map generating unit 104 generates a depth map. The depth map is for expressing depth information in a three-dimensional space. The depth map generating unit 104 acquires information on the distance to an object in real space or a CG content that is superimposed and displayed based on the user's viewpoint position, and creates a depth map. The distance information to the object in real space can be calculated, for example, from the parallax between two images captured by the imaging unit 102. The method of calculating the distance information from the two images can use existing technology. The method of generating the depth map is not limited to the above method, and other methods such as using LiDAR (Light Detection And Ranging) may be used.

[0024] The pointer position calculation unit 105 calculates the position pointed to by the pointer. The position in the mixed reality space can be pointed to by using the operation device 120 that is associated with the HMD 100 in advance. The operation device 120 will be described in detail later. When the user uses the operation device 120 to perform an operation of pointing to an object in the real space or CG, the pointer position calculation unit 105 calculates the user's pointing direction from the position and orientation of the operation device 120 acquired via the device communication unit 107. The pointer position calculation unit 105 specifies the pointing direction calculated from the position and orientation information of the operation device 120 and the position pointed to in the three-dimensional space from the depth map described above.

[0025] The determination unit 112 determines the visibility of the other user's pointed position when viewed from the user's viewpoint. Specifically, the determination unit 112 determines whether or not the other user's pointer position is in a position visible from the user's viewpoint, based on the pointer position information of the other user obtained via the server communication unit 108 and the depth map generated by the depth map generation unit 104. The UI display control unit 106 generates information on the method of displaying the pointer and ray according to the determination result of the determination unit 112.

[0026] The device communication unit 107 performs wireless communication with the operation device 120. Through wireless communication, the HMD 100 acquires operation information of buttons and the like of the operation device 120 and sensor information mounted on the operation device 120. For communication with the operation device 120, Bluetooth (registered trademark), wireless LAN, or the like is used.

[0027] The server communication unit 108 communicates with the server. Wireless LAN or the like is used for communication with the server. In this embodiment, it is assumed that a plurality of users gather at the same place in the real space to join (connect to) the server, and share one mixed reality space with the plurality of users. The server communication unit 108 transmits and receives necessary information such as position information of other participating users via the server. In addition, in order to display the designated position of other users with a pointer when a plurality of people work in the mixed reality space, the HMD 100 transmits position and orientation information of its own operating device 120, the designated position, and information of the operation to the server, and receives information of other users from the server.

[0028] The image generation unit 109 generates a composite image representing a mixed reality space by combining an image acquired from the imaging unit 102 with content such as CG. The viewpoint of the CG is determined by acquiring position and orientation information estimated by the position and orientation estimation unit 103. In the first embodiment, an example of generating a composite image representing a mixed reality space will be described, but an image representing a virtual reality space composed only of CG may be generated. Furthermore, the image generation unit 109 combines CG of a pointer and a ray according to operation device information acquired via the device communication unit 107 and information generated by the UI display control unit 106.

[0029] The image display unit 110 displays the image generated by the image generation unit 109. The image display unit 110 has, for example, a liquid crystal panel or an organic EL panel. When the user wears the HMD 100, the image display units 110 are disposed at the right and left eyes of the user.

[0030] The memory 111 is a storage medium that holds various data necessary for performing processing within the HMD 100. Examples of the data held in the memory 111 include user information and designated position information acquired by the server communication unit 108, and sensor information of the operation device 120 received by the device communication unit 107.

[0031] In this embodiment, an example in which the present invention is applied to a head-mounted HMD 100 will be described, but the configuration of the present invention is not limited to an HMD. For example, the present invention may be applied to a personal computer, a smartphone, a tablet terminal, etc. equipped with a display and a camera. In addition, in this embodiment, an information processing unit (information processing device) that handles image processing and information processing is built into the HMD 100, but the information processing unit (information processing device) may be separate from the HMD 100.

[0032] (Configuring the operation device) Next, the internal configuration of the operation device 120 will be described with reference to Fig. 1. The operation device 120 is a device for a user to input instructions (commands) to the HMD 100, and can also be said to be a control device for controlling the HMD 100 through user operations. The operation device 120 has a device control unit 121, an operation unit 122, a communication unit 123, and an inertial sensor 124.

[0033] The device control unit 121 controls each component of the operation device 120. The operation unit 122 is an operation unit such as a button operated by a user. The communication unit 123 transmits operation information of the operation unit 122 and sensor information acquired by the inertial sensor 124 to the HMD 100 by wireless communication. The inertial sensor 124 is an inertial measurement unit (IMU) and acquires three-dimensional angular velocity and acceleration as sensor information. The inertial sensor 124 may further include a geomagnetic sensor and multiple angular velocity sensors.

[0034] The operation device 120 is also called a "hand controller" or simply a "controller." A controller that is shaped to be held by the user's hand is also called a grip type controller or handheld type controller, and a controller that is used while being worn on the user's hand or finger is also called a wearable type controller. In this embodiment, for example, as shown in FIG. 2A and FIG. 2B, a finger ring type operation device 120 is used so that it can be worn on the user's finger. If the operation device 120 can be worn on the user's finger, there is an advantage that the user can freely move his or her hand or fingers while holding the operation device 120, and the operation device 120 is less likely to cover the hand.

[0035] Although the shape of the operation device 120 is a ring type, the shape is not limited to this. For example, the shape of the operation device 120 may be a glove type that can be worn on the hand, or a hand-held type. The operating device 120 may have a shape that can be worn on the wrist, such as a wristwatch type (bracelet type). In this way, the operating device 120 may be in a form that can be held by the user's hand or worn on the hand or wrist, so that the user can easily use it. A plurality of operating devices for operating the HMD 100 may be provided. For example, an operating device for the right hand and an operating device for the left hand may be provided, or the operating devices may be worn on a plurality of fingers (for example, the thumb and index finger, etc.).

[0036] The operation unit 122 may be configured with any operation member that is operated by the user through physical contact. For example, the operation unit 122 may have an OTP (optical track pad) that can sense the amount of planar movement. The operation unit 122 may also include any of a touch pad, a touch panel, a cross key, a button, a joystick, and a track pad device. Alternatively, if only changes in the position and / or attitude of the operation device 120 itself are used as the operation by the operation device 120, the operation unit 122 may not be necessary.

[0037] (About Ray Display) Referring to FIG. 3, pointer operations using the operation device 120 will be described.

[0038] As indicated by reference numeral 302 in FIG. 3, an operation device coordinate system (xyz Cartesian coordinate system) based on the position and orientation of the operation device 120 is defined. The HMD control unit 101 receives sensor data acquired by the inertial sensor 124 from the operation device 120 via the device communication unit 107 and the communication unit 123, and calculates the orientation of the operation device 120 based on the sensor data. A known technique can be used to calculate the orientation of the operation device 120. The position of the operation device 120 is specified by a method such as specifying the position of the operation device 120 by image recognition using an image obtained by the imaging unit 102 of the HMD 100. A known technique such as machine learning can be used as the method of specifying the position by image recognition.

[0039] The HMD 100 stores a setting value of the pointing direction of the operation device 120. For example, as shown in Fig. 3, a pointing direction 303 is set parallel to the x-axis (negative direction) of the operation device coordinate system. This allows the user to operate the operation device 120 to point to a distant position according to the movement of the hand wearing the operation device 120.

[0040] In the first embodiment, in order to allow the user to easily recognize the position of the pointer, the image generating unit 109 synthesizes a ray extending from the operation device 120 along the pointing direction 303 with a CG of a pointer indicating the pointed position. The ray is a CG object that extends linearly along the pointing direction 303 as if a light beam were being emitted from the operation device 120. The pointer is a CG object that represents the tip of the ray (the intersection of the ray and an object (an object in real space or a virtual object created by CG)).

[0041] (About displaying other users' pointers and rays) In the first embodiment, participating users each wear an HMD 100 and an operation device 120 in the information processing system 1, and communicate with each other via a server. In this situation, when each user points to an object in the mixed reality space, a pointer and a ray are displayed so that other participating users can see the point being pointed to.

[0042] The processing of the determination unit 112 and the UI display control unit 106 regarding control of the display of the pointer and ray will be described in detail with reference to the flowchart of FIG. In step S401, the determining unit 112 obtains from the memory 111 the depth map generated by the depth map generating unit 104 based on the current viewpoint position of the user.

[0043] The subsequent processing of steps S402 to S405 is executed for each of the other participating users. In the following explanation, the participating user selected as the processing target is referred to as the "target user." It is called. In step S402, the determination unit 112 reads data of the target user previously acquired via the server communication unit 108 from the memory 111, and acquires information on the position the target user is pointing to (hereinafter referred to as the "other person's pointed position").

[0044] In step S403, the determination unit 112 checks whether the other person's designated position is in a position that can be seen from the user's viewpoint. If the other person's designated position is in a position that can be seen from the user's viewpoint, the determination unit 112 determines that the visibility of the other person's designated position is good, and if the other person's designated position is in a position that cannot be seen from the user's viewpoint, the determination unit 112 determines that the visibility of the other person's designated position is bad. Specifically, the determination unit 112 calculates a projection position when the other person's designated position acquired in step S402 is projected onto a screen on which the user's field of view is displayed. Then, the determination unit 112 compares the calculated projection position with the depth information of the corresponding position in the depth map acquired in step S401 to determine whether the other person's designated position is in front of or behind (in a blind spot) an object present in the user's field of view. If the other person's designated position is in front of the object, the determination unit 112 determines that the other person's designated position is in a position that can be seen (good visibility), and if the other person's designated position is in a position that cannot be seen (poor visibility). In this way, the depth map can be used to determine the visibility of the other person's pointed position.

[0045] In step S404, the UI display control unit 106 performs a display setting of the pointer. If it is determined in step S403 that the other person's designated position is a position visible from the user's viewpoint, the UI display control unit 106 performs a setting to display the pointer, and otherwise performs a setting to not display the pointer. By hiding the pointer when the other person's designated position is a position invisible to the user, it is possible to prevent other users from misidentifying the designated position. Furthermore, the UI display control unit 106 performs a setting regarding the display method of the pointer. The setting of the display method of the pointer includes, for example, setting the color and shape of the pointer, and setting the text (annotation) to be displayed beside the pointer. By changing the display method of the color, shape, text, etc. for each user, the user can distinguish the pointer by the pointer. In this embodiment, the pointer is not displayed when the other person's designated position is a position invisible to the user, but the control is not limited to this. For example, the pointer may be displayed in a different way from the normal pointer display, such as by displaying the pointer semi-transparently, to express that the invisible position is being designated. In addition, when the pointer overlaps with the ray due to the ray display, the color and size may be set to be emphasized so that the pointer is easily visible.

[0046] In step S405, the UI display control unit 106 sets the display of rays. If the pointing position of the target user is visible from the user's viewpoint, the UI display control unit 106 sets the target user's rays not to be displayed. This makes it possible to prevent a decrease in visibility, such as making it difficult to see CG content due to an increase in rays on the screen. Note that, since the pointing position of the target user can be recognized by the pointer due to the pointer display setting in step S404, there is no major problem even if the target user's example is not displayed. On the other hand, if the pointing position of the target user is not visible from the viewpoint of the user, the UI display control unit 106 sets the display of a ray. In this way, it becomes possible to grasp the approximate pointing direction with respect to a pointing position that is not visible to the user.

[0047] In step S406, it is determined whether the processes from step S402 to step S405 have been performed for each user other than the user himself / herself, and if the processes for all users have not been completed, an unprocessed user is selected as the target user, and the process returns to step S402. When the processes for all users have been completed, the process ends.

[0048] (Other ways to display pointers and rays) When the pointing position of another user is near the boundary between the area visible from the user's viewpoint and the area invisible, the pointing position of another user may frequently cross the boundary between the area visible and the area invisible due to shaking of the other user's hand or shaking of the user's viewpoint. In such a case, in the basic operation of the UI display control unit 106 described above, switching between the display of the pointer and the display of the ray occurs frequently, resulting in reduced visibility and visual annoyance. In order to solve such a problem, the UI display control unit 106 may switch the display method of the pointer or the ray of another user when the same determination result continues for a predetermined time after the determination result by the determination unit 112 has changed.

[0049] For example, in the processing of steps S404 and S405, even if the position pointed to by another person moves from a position invisible to the user to a position visible to the user, the display may not be changed and the ray may remain hidden for a predetermined time (for example, several hundred milliseconds to several seconds). Furthermore, the display method settings in the processing of steps S404 and S405 are not limited to those described above, and as a display method near the boundary, a display setting may be made such that both the ray and the pointer are displayed for a predetermined time after crossing the boundary. The length of this predetermined time may be set by the user.

[0050] Furthermore, it may be set so that an instruction UI such as a ray or a pointer is displayed only when a user is performing a predetermined operation, such as displaying a ray only while the user is operating the operation unit 122 of the operation device 120. This allows the user to switch to a display state of other users' rays or pointers by performing a predetermined operation when the user wants to check the pointing position or pointing direction of other users, and to hide the ray or pointer in other cases to increase the visibility of objects in the field of view.

[0051] (Other ways to display pointers and rays 2) When multiple people display rays, the rays may overlap each other or the rays may overlap the pointer, which may reduce the visibility of the rays and the pointer. For example, in a situation where multiple rays 50 are displayed overlapping each other as shown in Figure 5, an object 51 displayed behind the rays 50 may become difficult to see near the overlapping area.

[0052] In order to solve such a problem, the UI display control unit 106 may change the display method when rays or pointers overlap. Specifically, the determination unit 112 acquires information on the position and the pointing position of the operation device 120 of each user, and determines whether rays are displayed overlapping based on the information. If the result of the determination is that a predetermined number or more of rays are displayed overlapping, the UI display control unit 106 changes the display method of rays so as to increase the transparency of the rays when composited with other CG. Here, the "predetermined number" may be set to any value of 2 or more. The change in the display method of rays is not limited to the change in transparency, and other methods such as displaying rays thinly may be used. Whether rays are displayed overlapping can be determined by calculating the coordinates when the position and the pointing position of each user's operation device 120 are projected onto the screen, and checking whether the lines connecting the operation device 120 and the pointing position intersect with each other.

[0053] In addition, visibility may be reduced in a situation where the pointer displayed at the indicated position overlaps with another user's ray. When the pointer overlaps with another user's ray, the UI display control unit 106 may set the indicated position to be easily recognized by changing the size of the pointer, highlighting the color of the pointer, or the like.

[0054] (3 other ways to display pointers and rays) In the first embodiment, an example has been described in which participating users gather in the same place to perform operations, but a user may participate remotely from a remote location. Each remote user is mapped to a specific position in the mixed reality space or virtual space. In such a situation, the position of the remotely participating user cannot be directly seen by other participating users, so it is advisable to display a CG (such as an avatar) at the user's position based on the user information acquired via the server communication unit 108 so that other users can see the user.

[0055] Furthermore, since it is better to know the starting point of the operation when displaying a pointer or ray so that other users can see the indicated position, it is advisable to display a CG of the operation device 120 at a predetermined position on the user's hand.

[0056] Furthermore, in the processing of the UI display control unit 106, the correspondence between the user and the pointer may be made clear by using display techniques such as making the colors of the CG of the operation device 120 and the pointer the same and changing the color for each user.

[0057] As described above, in the first embodiment, the display method of the pointers and rays of other participating users is changed depending on whether the pointing positions of other participating users are visible from the viewpoint position of the user. Therefore, it is possible to provide a display method that allows the pointing positions of other users to be easily grasped regardless of their pointing positions and does not reduce visibility.

[0058] [Second embodiment] In the first embodiment, whether or not the pointed position is visible is determined based on the depth map of an object in real space or CG content displayed in a superimposed manner, and the context of the pointed position of another user, and the method of displaying the pointer or ray is changed. In the second embodiment, an example of an information processing device that changes the representation of the pointer or CG content at the pointed position depending on the distance between the user's viewpoint and the pointed position of another user will be described.

[0059] First, a configuration example of an information processing system according to the second embodiment will be described with reference to the block diagram of Fig. 6. The information processing system 1 according to this embodiment has, as hardware, an HMD 700 and an information processing device 600. The HMD 700 and the information processing device 600 are connected to each other via wire and / or wirelessly so that they can communicate data with each other.

[0060] The HMD 700 will be described. A user who is an observer can observe a virtual reality space or a mixed reality space through the HMD 700 by wearing the HMD 700 on his / her head. Note that, although the HMD 700 is given as an example of a head-mounted display device in this embodiment, other types of head-mounted display devices may be applied. In addition to the head-mounted display device, other types of display devices, such as a handheld display device, may be applied as long as they are a display device viewed by an observer to allow the observer to observe a virtual reality space or a mixed reality space.

[0061] The display unit 702 displays an image of the virtual reality space or mixed reality space sent from the information processing device 600. The display unit 702 may be configured to include two displays arranged corresponding to the left and right eyes of the observer, respectively. In this case, an image of the virtual reality space or mixed reality space for the left eye is displayed on the display corresponding to the left eye of the observer, and an image of the virtual reality space or mixed reality space for the right eye is displayed on the display corresponding to the right eye of the observer.

[0062] The imaging unit 701 captures moving images in real space, and has an imaging unit (right imaging unit) 701R that captures images to be provided to the right eye of the observer, and an imaging unit (left imaging unit) 701L that captures images to be provided to the left eye of the observer. Images of each frame (images of real space) constituting the moving images captured by the imaging units 701R and 701L are sequentially sent to the information processing device 600. In the case of a system for observing a virtual reality space, the imaging unit 701 captures images of the real space. The information processing device 600 does not need to transmit the moving image between the right and left imaging units 701R and 701L to the information processing device 600. In order to calculate the position and orientation of the imaging unit 701, an imaging unit (imaging unit for position and orientation calculation) 701N that captures moving images in real space different from the right imaging unit 701R and the left imaging unit 701L may be provided. The relative positional relationship between the imaging unit for position and orientation calculation 701N and the right and left imaging units 701R and 701L is held in advance in the information processing device 600. The internal parameters (focal length, principal point, angle of view, etc.) of the imaging units 701N, 701R, and 701L are also held in advance in the information processing device 600.

[0063] The measuring unit 703 functions as a receiver of the magnetic field sensor system and measures its own position and orientation. The magnetic field sensor system will be described with reference to FIG. 7. The magnetic field generating device 801 functions as a transmitter in the magnetic field sensor system and is fixedly disposed at a predetermined position in real space, generating a magnetic field around itself. The operation of the magnetic field generating device 801 is controlled by a controller 802, and the operation of the controller 802 is controlled by the information processing device 600.

[0064] The measuring unit 703 is fixedly attached to the HMD 700, measures a change in the magnetic field generated by the magnetic field generating device 801 according to its own position and orientation in the magnetic field, and transmits the measurement result to the controller 802. The controller 802 generates a signal value indicating the position and orientation of the measuring unit 703 in a sensor coordinate system 804 from the measurement result, and transmits the signal value to the information processing device 600. The sensor coordinate system 804 is a coordinate system (x, y, z) in which the position of the magnetic field generating device 801 is set as the origin, and three axes perpendicular to each other at the origin are the x-axis, y-axis, and z-axis. In this embodiment, the position and orientation of the user (HMD 700) are detected by a magnetic sensor system, but instead of the magnetic sensor system, an ultrasonic sensor system or an optical sensor system may be used, or these systems may be used in combination.

[0065] The information processing device 600 will be described. The information processing device 600 is composed of a computer device such as a PC (personal computer) or a portable terminal device such as a smartphone or a tablet terminal device. The information processing device 600 has an acquisition unit 601, an estimation unit 602, a three-dimensional information generation unit 603, a calculation unit 604, a transmission / reception unit 605, a storage unit 606, a determination unit 607, a UI display control unit 608, and an image generation unit 609 as main functional units.

[0066] FIG. 8 is a diagram showing a basic configuration of a computer that can be used as the information processing device 600 according to this embodiment. In FIG. 8, a processor 901 is, for example, a CPU, and controls the operation of the entire computer. A memory 902 is, for example, a RAM, and temporarily stores programs and data. A computer-readable storage medium 903 is, for example, a hard disk or a solid state drive, and non-temporarily stores programs and data. In this embodiment, a program that realizes the functions of each unit stored in the storage medium 903 is read into the memory 902. Then, the processor 901 operates according to the program on the memory 902, thereby realizing the functions of each functional unit described below. In addition, the input I / F 905 inputs an input signal from an external device in a format that can be processed by the information processing device. In addition, the output I / F 906 outputs an output signal to an external device in a format that can be processed.

[0067] The operation of each functional unit of the information processing device 600 of this embodiment will be described with reference to the flowchart in Fig. 9. Through the processing of the flowchart in Fig. 9, the information processing device 600 transmits its own pointed position to other users. Meanwhile, the information processing device 600 receives the pointed position of the other users and changes the representation of the pointed position and CG content according to the distance from its own viewpoint. This makes it easy to grasp the pointed position of other users while preventing a decrease in visibility.

[0068] In S1001, the acquisition unit 601 acquires an image of the imaging unit 701 and a calculation in the sensor coordinate system 804. The position and orientation of the imaging unit 701 is received from the imaging unit 701. In the case of a system observing a virtual reality space, only the image from the imaging unit 701N for position and orientation calculation may be received. In the case of a system observing a mixed reality space, it is preferable to receive an image from the right imaging unit 701R, an image from the left imaging unit 701L, and an image from the imaging unit 701N for position and orientation calculation.

[0069] In S1002, the estimation unit 602 estimates the position and orientation of the right and left viewpoints of the HMD 700 in a world coordinate system 803, with the right imaging unit 701R as the right viewpoint and the left imaging unit 70L as the left viewpoint. The world coordinate system 803 is an orthogonal coordinate system (X, Y, Z) with the origin being a reference point set in a real space in which a user (observer) exists. Conversion information for converting the position and orientation in the sensor coordinate system 804 into the position and orientation in the world coordinate system 803 is obtained in advance and is registered in the information processing device 600 in advance. In addition, the relative positional relationship (right eye bias) between the measurement unit 703 and the right imaging unit 701R and the relative positional relationship (left eye bias) between the measurement unit 703 and the left imaging unit 701L are also obtained in advance and are registered in the information processing device 600 in advance.

[0070] Specifically, the estimation unit 602 acquires a signal value indicating the position and orientation of the measurement unit 703 in the sensor coordinate system 804 from the measurement unit 703 (via the controller 802 in the case of FIG. 7). Next, the estimation unit 602 converts the position and orientation indicated by the signal value into a position and orientation in the world coordinate system 803 using the above conversion information. Then, the estimation unit 602 estimates the position and orientation of the right viewpoint in the world coordinate system 803 by adding a bias for the right eye to the converted position and orientation. Similarly, the estimation unit 602 calculates the position and orientation of the left viewpoint in the world coordinate system 803 by adding a bias for the left eye to the converted position and orientation. In the following, in a description common to the right viewpoint and the left viewpoint, the right viewpoint and the left viewpoint may be collectively referred to simply as viewpoints.

[0071] Note that various other methods can be applied to the method for determining the position and orientation of the viewpoint in the world coordinate system 803. For example, a marker (also called an AR marker) assigned to the world coordinate system 803 is extracted from an image in real space. Then, the position and orientation of the viewpoint in the world coordinate system 803 is calculated based on the position and orientation of the marker. The position and orientation of the position and orientation calculation imaging unit 701N may extract a marker and determine the position and orientation of the position and orientation calculation imaging unit 701N in the world coordinate system 803 based on the position and orientation of the marker, and then the viewpoint may be calculated based on the relative positional relationship between the right imaging unit 701R and the left imaging unit 701L. Alternatively, the position and orientation of the viewpoint may be determined by performing Simultaneous Localization and Mapping (SLAM) processing based on feature points captured in an image in real space.

[0072] In S1003, the three-dimensional information generating unit 603 generates three-dimensional information from the image of the imaging unit 701 acquired by the acquisition unit 601 and the position and orientation of the imaging unit 701 estimated by the estimation unit 602. The three-dimensional information is a polygon having three-dimensional position information in a world coordinate system 803. For example, the three-dimensional information generating unit 603 can calculate the depth of each pixel from parallax information in stereo images acquired from the right imaging unit 701R and the left imaging unit 701L. The three-dimensional information generating unit 603 generates a three-dimensional point group in the world coordinate system 803 from the depth information of each pixel and the position and orientation of the imaging unit 701, and generates a polygon by connecting the point group.

[0073] In S1004, the calculation unit 604 calculates its own pointing position. A method for calculating the pointing position will be described with reference to FIG. 10. Reference numeral 1101 denotes a three-dimensional orthogonal coordinate system (Xe, Ye, Ze) of the viewpoint of the HMD 700. The viewpoint coordinate system 1101 has the viewpoint 1100 as its origin, the Ze axis parallel to the optical axis of the imaging unit 701, and the Xe and Ye axes parallel to the image plane. In this embodiment, the upward direction of the HMD 700 is set as the Ye positive direction, the direction of the user's right hand is set as the Xe positive direction, and the direction opposite to the line of sight is set as the Ze positive direction. The viewpoint 1100 is placed at the position of the right or left viewpoint, or at the center between the right and left viewpoints. The object 1102 is determined by the three-dimensional information generation unit 6 11 indicates the three-dimensional information (real object) generated in 1103, or the CG content (virtual object) acquired from the storage unit 606. The calculation unit 604 calculates, for example, the intersection of a vector in the negative direction of the Ze axis of the viewpoint coordinate system 1101 and a polygon of the object 1102 as the "pointed position". The calculation unit 604 also calculates a surface normal vector 1103 of the polygon at the pointed position. In this manner, the calculation unit 604 calculates its own pointed position. The CG content acquired from the storage unit 606 here includes information necessary for drawing the CG content, such as polygon information, which is shape information of the CG content in the world coordinate system 803, color information, information expressing the texture, and the like, information defining the texture, and the like.

[0074] In this embodiment, the user's viewpoint and line of sight direction are detected to identify a point (point of gaze) in the virtual space on which the user is gazing, and a UI such as a ray indicating the line of sight direction (i.e., the pointing direction) and a pointer indicating the point of gaze (i.e., the pointing position) is displayed. However, the method in which the user inputs the pointing direction and pointing position, or the method in which the user's pointing direction and pointing position are calculated, are not limited to this. For example, the operation device described in the first embodiment may be used. That is, the user's pointing direction may be detected by detecting the position and orientation of the operation device worn or held by the user, and the intersection of the vector of the pointing direction and the object 1102 may be calculated as the pointing position. Alternatively, the user's hand may be detected, and the intersection of the vector of the direction of the index finger and the object 1102 may be calculated as the pointing position. The HMD 700 may detect the user's line of sight, and calculate the intersection of the vector of the user's line of sight direction and the object 1102 as the pointing position. Alternatively, a marker attached to an object held by the user can be detected by a sensor such as a fixed camera, the vector of the user's pointing direction can be identified from the position and orientation of the marker, and the intersection of that vector with the object 1102 can be calculated as the pointing position.

[0075] In S1005, the transmitting / receiving unit 605 transmits the user's designated position, a surface normal vector at the designated position, and the position and orientation of the viewpoint to the information processing device of the other user. The user may be allowed to set in the information processing device 600 whether or not to transmit the user's information to other users, and to select a user to be the transmission destination. In this case, it is also preferable that the setting in the information processing device 600 can be operated using the operation device 120 described in the first embodiment.

[0076] In S1006, the transmission / reception unit 605 receives the pointing position, surface normal vector, and viewpoint position and orientation of the other user from the information processing device of the other user.

[0077] In S1007, the determination unit 607 calculates the distance between the position of the user's viewpoint and the pointing position of the other user. Then, the determination unit 607 compares the calculated distance with a threshold. If the distance between the user's viewpoint and the pointing position of the other user is equal to or greater than the threshold, it is determined that the visibility of the pointing position of the other user from the user's viewpoint is poor (not visible or difficult to see). If the distance is equal to or less than the threshold, it is determined that the visibility of the pointing position of the other user from the user's viewpoint is good (visible or easy to see). The threshold may be a fixed value or a value that the user himself / herself has set in advance in the information processing device 600. Alternatively, the determination unit 607 may change the threshold adaptively (dynamically). Furthermore, the determination unit 607 may change the threshold for each other user (i.e., for each pointing position). For example, the distance between the position of the viewpoint of the other user and the pointing position of the other user itself, or a value obtained by multiplying the distance by a coefficient, may be used as the threshold. The threshold may also be dynamically determined based on the distance between your viewpoint and the viewpoint of another user, the distance between your viewpoint and your pointing position, the distance between your pointing position and the pointing position of another user, the size of the object 1102, the direction of the surface normal, etc.

[0078] In S1008, the UI display control unit 608 receives the result of the determination unit 607 and changes the representation of the pointed position or the representation of the object 1102. If the pointed position of another user is not visible or is difficult to see, the UI display control unit 608 may improve the visibility (visual distinctiveness) of the pointed position. At this time, the UI display control unit 608 may improve the visibility (visual distinctiveness) of the pointed position so as to make the pointed position more visually noticeable than the object 1102. In other words, the visibility of the pointed position is prioritized over the object 1102. On the other hand, when the pointed position of another user is visible or easily visible, the UI display control unit 608 may change the representation of both or one of the pointed position and the object 1102 so as to suppress a decrease in the visibility of the object 1102 due to the display of the pointed position. In other words, the visibility of the object 1102 is prioritized over the pointed position.

[0079] The change in the indication position and the expression of the object 1102 will be described with reference to FIG. 11A and FIG. 11B. FIG. 11A shows an example in which the distance L between the user's viewpoint 1100 and the indication position 1200 of another user is determined to be equal to or greater than a threshold. When the distance L is equal to or greater than a threshold, the indication of the pointer 1201 representing the indication position 1200 is emphasized in order to improve the visibility of the indication position 1200 of the other user. Specifically, the size of the pointer 1201 is made larger than normal. In addition, the pointer 1201 may be made in an emphasized color, or an emphasis effect such as blinking or animation may be added. In addition, an annotation 1202 may be added, or the object 1102 overlapping with the pointer 1201 may be made semi-transparent. FIG. 11B shows an example in which the distance L between the user's viewpoint 1100 and the indication position 1200 of the other user is determined to be less than a threshold. When the distance L is less than the threshold, for example, the pointer 1201 at the pointed position 1200 is made small and only a frame is displayed in order to prioritize the visibility of the object 1102. Alternatively, the pointer 1201 at the pointed position 1200 may be made semi-transparent. By changing the representation of the pointer 1201 and / or the representation of the object 1102 according to the distance L as described above, it is possible to achieve both ease of grasping the pointed position 1200 of other users and prevention of a decrease in the visibility of the object 1102.

[0080] It should be noted that the above-mentioned changes in the representation of the pointer 1201 and the object 1102 may be all implemented, or at least one of them may be implemented. Of course, other changes in representation may be used. In addition, in this embodiment, the display is switched depending on whether the distance L is equal to or greater than a threshold value, but the display may be switched in three or more stages depending on the distance L. For example, the pointer 1201 may be made larger in stages as the distance L increases, or the transparency of the object 1102 may be increased. Conversely, the pointer 1201 may be made smaller in stages as the distance L decreases, or the transparency of the pointer 1201 may be increased in stages.

[0081] In S1009, the image generation unit 609 first constructs a virtual space in which each CG content is arranged, using the CG content acquired from the storage unit 606 and the pointer and CG content information of the designated position changed by the UI display control unit 608. The image generation unit 609 then generates images of the virtual space seen from the positions and orientations of the right and left viewpoints calculated in S1002, also using the internal parameters of the imaging unit 701. As a known technique can be used to generate images of the virtual space seen from the viewpoints, a description thereof will be omitted.

[0082] The image generating unit 609 generates an image of the mixed reality space by synthesizing the generated image of the virtual space with the image of the real space acquired from the imaging unit 701. The synthesis process is performed by superimposing the image of the virtual space on the image of the real space. That is, the pixels other than the area of ​​the CG content are the pixels of the real space displayed in the synthesized image. At this time, in order to improve the visibility of the pointer 1201 at the indicated position 1200, the color or brightness of the pixels of the object in the real space on which the pointer 1201 is superimposed may be changed. When observing the virtual space, a video signal of the virtual space image for the right eye is sent to the display for the right eye, and a video signal of the virtual space image for the left eye is sent to the display for the left eye. When observing the mixed reality space, an image of each mixed reality space is sent to each display instead of each virtual space image. This allows a user (observer) wearing the HMD 700 to observe the virtual space or the mixed reality space.

[0083] In S1010, it is determined whether the user has input an instruction to end the process. If an instruction to end the process is input, the process ends in the flowchart of Fig. 9. If an instruction to end the process is not input, the process returns to S1001.

[0084] As described above, according to this embodiment, by changing the representation of the pointer or CG content of the pointed position depending on the distance between one's own viewpoint position and the pointed position of another user, it is possible to achieve both ease of grasping the pointed position of another user and prevention of a decrease in visibility of objects.

[0085] (Modification 1 of the second embodiment) In the above-described second embodiment, the pointer and CG representation of the pointed position are changed according to the distance between the user's viewpoint and the pointed position of another user. In this modified example, the pointer and CG representation of the pointed position are changed based on the relationship between the orientation of the surface of the pointed position of another user and the user's viewpoint and line of sight. In the following, the description of the parts common to the second embodiment will be omitted as appropriate, and the description will focus on the differences. In the system according to this modified example, the operation of the determination unit 607 and the UI display control unit 608 is different from that of the second embodiment.

[0086] The determining unit 607 determines the visibility of the other user's pointed position from the user's viewpoint based on the position and orientation (line of sight direction) of the user's viewpoint and the other user's pointed position and its surface normal vector.

[0087] An example of the determination method will be described with reference to Figures 12A and 12B. First, the determination unit 607 determines whether the position of the viewpoint 1100 is on the front side or the back side of the surface 1301 calculated from the surface normal vector 1103, based on the position of the user's viewpoint 1100, the other user's designated position 1200, and the surface normal vector 1103. The direction in which the surface normal vector 1103 faces is the front side of the surface 1301. Next, the determination unit 607 calculates the angle θ between the vector 1100z in the positive direction of the Ze axis of the viewpoint coordinate system and the vector 1203 pointing from the designated position 1200 to the position of the user's viewpoint 1100.

[0088] 12A shows a case where the viewpoint 1100 is on the front side of the surface 1301. In this case, if the angle θ is between -90 degrees and 90 degrees (the absolute value of θ is less than 90 degrees), it is determined that the visibility of the pointed position 1200 of another user from the viewpoint 1100 is good (visible or easily visible). If the angle θ is outside the angle range of -90 degrees to 90 degrees (the absolute value of θ is 90 degrees or more), it is determined that the visibility of the pointed position 1200 from the viewpoint 1100 is poor (not visible or difficult to see).

[0089] 12B shows a case where the viewpoint 1100 is on the back side of the surface 1301. In this case, if the angle θ is between -90 degrees and 90 degrees (the absolute value of θ is less than 90 degrees), it is determined that the other user's pointed position 1200 is within the user's field of view, but the pointed position 1200 is not visible due to occlusion by the object 1102. If the angle θ is outside the angle range of -90 degrees to 90 degrees (the absolute value of θ is 90 degrees or more), it is determined that the pointed position 1200 is not visible from the user's viewpoint 1100.

[0090] In this example, the angle range (threshold) of the field of view is set to ±90 degrees, but this angle range may be set arbitrarily. For example, the angle range may be set based on the user's field of view (for example, the narrower of the vertical angle of view and the horizontal angle of view of the imaging unit 701). The angle range may be a fixed value or a value that the user himself / herself has set in advance in the information processing device 600. Alternatively, the determination unit 607 may adaptively (dynamically) change the angle range. Furthermore, the determination unit 607 may change the angle range for each other user (i.e., for each pointing position). For example, the angle range may be dynamically determined depending on the distance between the user's viewpoint 1100 and the pointing position 1200 of the other user. Alternatively, instead of changing the angle range (threshold serving as the basis for determination) depending on the distance, the determination result may be changed depending on the distance. In other words, if an object is within a predetermined angle range and is "visible or not visible," the determination unit 607 may determine whether the object is "visible" or "not visible." Even if it is determined that the object is "easy to see," if the distance is equal to or greater than a threshold, the determination result may be changed to "not visible or difficult to see."

[0091] The UI display control unit 608 changes the representation of the pointed position or the representation of the object 1102 according to the determination result of the determination unit 607. Specifically, when it is determined that the pointed position 1200 is visible or easily visible, the UI display control unit 608 may change the representation of both or one of the pointed position 1200 and the object 1102 so as to suppress a decrease in visibility of the object 1102 due to the display of the pointed position 1200. For example, the pointer of the pointed position 1200 may be made smaller or may be displayed as just a frame. Alternatively, the pointer of the pointed position 1200 may be made semi-transparent.

[0092] When it is determined that the pointed position 1200 is within the field of view but is obscured by the object 1102, the UI display control unit 608 may make the object 1102 semi-transparent so that the pointer at the pointed position 1200 is visible. Alternatively, an annotation may be added to the pointed position 1200 as shown in Fig. 11A to inform the user that the pointer at the pointed position 1200 is hidden behind the object 1102. Alternatively, a message "Another user is pointing to the back side of the object" may be displayed on the display screen of the HMD 700 to prompt the user to move the viewpoint.

[0093] When it is determined that the pointed position 1200 is within the field of view but is difficult to see because it is far away from the viewpoint, the UI display control unit 608 may emphasize the representation of the pointer representing the pointed position 1200 in order to improve the visibility of the pointed position 1200. Specifically, the size of the pointer may be made larger than normal, the pointer may be given an emphasized color, an emphasis effect such as blinking or animation may be added, an annotation may be added, or an object overlapped by the pointer may be made semi-transparent. In addition, the user's attention may be attracted by displaying a message such as "Another user is giving instructions" on the display screen of the HMD 700.

[0094] As described above, according to this modification, it is possible to determine whether the point pointed by another user is visible based on the relationship between the direction of the surface pointed by the other user and the user's viewpoint position or line of sight, and to change the pointer or CG representation of the pointed position according to the determination. This makes it possible to easily grasp the pointed position of other users while preventing a decrease in the visibility of objects.

[0095] (Modification 2 of the second embodiment) In the above-mentioned modified example 1 of the second embodiment, the pointer and CG representation of the pointed position are changed in consideration of the direction of one's line of sight and the direction of the surface of the pointed position of another user. In this modified example, the pointer and CG representation of the pointed position are changed depending on whether the pointed position of another user is captured by the imaging unit. In the following, the description of the parts common to the second embodiment and modified example 1 of the second embodiment will be omitted as appropriate, and the different parts will be described in detail. In the system related to this modified example, the operation of the determination unit 607 is different from that of the second embodiment and modified example 1 of the second embodiment.

[0096] The determination unit 607 determines whether or not the pointing position of the other user is within the field of view based on the position and orientation of the imaging unit 701 of the HMD 700, the internal parameters of the imaging unit 701, and the pointing position of the other user received by the transmission / reception unit 605. For example, the determination unit 607 converts the pointing position of the other user in the world coordinate system into position information of the camera coordinate system of the imaging unit 701. After that, the determination unit 607 converts the position information of the camera coordinate system into position information of the image coordinate system by performing perspective projection transformation using the internal parameters of the imaging unit 701. It is possible to determine whether or not the pointing position of the other user is within the field of view based on whether or not the position of this image coordinate system is within the range of the image captured by the imaging unit 701. Since a known technology can be used for determining whether or not the pointing position of the other user is within the field of view, detailed description thereof will be omitted. If the pointing position of the other user is not within the field of view, the determination unit 607 determines whether the pointing position of the other user is visible from the viewpoint of the user. It is determined that there is no

[0097] Next, in the same manner as described in the first modification of the second embodiment, it is determined whether the position of the user's viewpoint is on the front side or the back side of the plane (i.e., the plane including the pointing position of the other user) calculated from the plane normal vector. When the user's viewpoint is on the front side of the plane including the pointing position of the other user and the pointing position of the other user is within the field of view, the determination unit 607 determines that the pointing position of the other user is visible or easily visible from the user's viewpoint. When the user's viewpoint is on the back side of the plane including the pointing position of the other user and the pointing position of the other user is within the field of view, the determination unit 607 determines that the pointing position of the other user is within the field of view but is not visible because it is hidden by an object. Here, even if it is determined that the pointing position of the other user is visible, the determination result may be changed in consideration of the distance between the user's viewpoint and the pointing position of the other user, as described in the first modification of the second embodiment. For example, even if it is determined that the pointing position of another user is "visible," if the distance between the user's viewpoint and the pointing position of the other user is greater than a threshold, the result may be changed to "not visible" or "difficult to see due to distance."

[0098] The UI display control unit 608 changes the CG representation of the pointer or object at the indicated position according to the determination result of the determination unit 607. The specific method of change may be the same as that described in the second embodiment and the first modified example of the second embodiment, and therefore a detailed description thereof will be omitted.

[0099] As described above, according to this modification, it is possible to determine whether the pointed position of another user is visible depending on whether the pointed position of the other user is captured by the imaging unit 701, and to change the expression of the pointer of the pointed position or the CG content depending on the determination. This makes it possible to easily grasp the pointed position of another user while preventing a decrease in the visibility of objects.

[0100] (Modification 3 of the second embodiment) In the above-mentioned modified example 2 of the second embodiment, whether the pointing position of another user is visible is determined depending on whether the pointing position of another user is captured by the imaging unit 701, and the expression of the pointer of the pointing position and the CG content is changed depending on the determination. In this modified example, when the pointing position of another user is visible for a certain period of time, the expression of the pointer of the pointing position and the CG content is changed to avoid a decrease in visibility of the CG content. In the following, the description of the parts common to the second embodiment, modified example 1 of the second embodiment, and modified example 2 of the second embodiment will be omitted as appropriate, and the description will focus on the different parts. In the system related to this modified example, the operation of the determination unit 607 and the UI display control unit 608 is different from that of the second embodiment, modified example 1 of the second embodiment, and modified example 2 of the second embodiment.

[0101] The operation of each functional unit of the information processing device 600 in this modified example will be described with reference to the flowchart in Fig. 13. Fig. 13 is a flowchart showing the processing procedure of the information processing device 600 in this modified example. Steps common to Fig. 9 are given the same step numbers and their description will be omitted.

[0102] The processes of S1001 to S1007 are the same as those of the second embodiment (FIG. 9). In S1401, the determination unit 607 measures the time during which it is determined that the pointing position of another user is visible from the user's viewpoint. That is, the determination unit 607 calculates the time during which the same determination result is continuously obtained from the time when it was first determined that the pointing position of another user is visible from the user's viewpoint, and sends the time information to the UI display control unit 608.

[0103] In S1402, the UI display control unit 608 takes into consideration the time measured by the determination unit 607 in the process of changing the CG representation of the pointer at the designated position or the object. The control unit 608 improves the visibility of the object by making the pointer representing the pointed position less noticeable as the time during which the pointed position is visible becomes longer or when the time during which the pointed position is visible exceeds a predetermined threshold. For example, the UI display control unit 608 may gradually increase the transparency of the pointer according to the length of time during which the pointed position is visible. Alternatively, the UI display control unit 608 may gradually decrease the size of the pointer according to the length of time during which the pointed position is visible. The transparency may be gradually increased or the size may be gradually decreased until the pointer is completely invisible. Alternatively, an upper limit value of the transparency and a lower limit value of the size may be determined in advance so that the values ​​are not exceeded. The UI display control unit 608 may change the pointer to a determined transparency and / or a determined size when the time during which the pointed position is visible exceeds a predetermined threshold. The UI display control unit 608 may make the pointer invisible when the time during which the pointed position is visible exceeds a predetermined threshold.

[0104] For example, assume a scene in which two persons A and B share the same mixed reality space and are discussing while observing the same object (real object or virtual object). When one person A points to a position p on the object and starts explaining, the other person B recognizes the area of ​​interest of person A by a pointer superimposed on the position p on the object. Once the area of ​​interest is known, person B would like to observe the position p on the object and its surroundings in more detail, and would find the pointer superimposed on the object to be a nuisance. According to the display control of this modification, if the pointer of person A is visible for a certain period of time, it is deemed that there is a high probability that person B has recognized the position pointed to by person A, and the pointer of person A is made inconspicuous (or invisible). This makes it possible to achieve both the ease of grasping the position pointed to by person A and the prevention of a decrease in the visibility of the object.

[0105] <Other> Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist of the present invention. The configurations described in the first and second embodiments may be combined with each other (as long as no technical contradiction occurs).

[0106] For example, in the above embodiment, an example in which the present invention is applied to a non-transmissive or video see-through HMD has been described, but the present invention may also be applied to an optical see-through HMD. The present invention may also be applied to a display device that is not a head-mounted type, such as a handheld display device, a stationary display device, a display device for a computer or smartphone, a projector screen, a retinal projection display, etc.

[0107] The present invention also includes a case where a software program for implementing the functions of the above-mentioned embodiment is supplied to a system or device having a computer capable of executing the program directly from a recording medium or by using wired / wireless communication, and the program is executed. Therefore, the program code itself that is supplied and installed to the computer in order to implement the functional processing of the present invention by the computer also realizes the present invention. In other words, the computer program itself for implementing the functional processing of the present invention is also included in the present invention. In that case, as long as it has the function of the program, the form of the program does not matter, such as object code, a program executed by an interpreter, script data supplied to an OS, etc. Examples of recording media for supplying the program include hard disks, magnetic recording media such as magnetic tapes, optical / magneto-optical storage media, and non-volatile semiconductor memories. In addition, as a method of supplying the program, a method in which a computer program forming the present invention is stored in a server on a computer network and a connected client computer downloads and executes the computer program is also considered. The present invention can also be realized by a process in which a program for implementing one or more functions of the above-mentioned embodiment is supplied to a system or device via a network or a storage medium, and one or more processors in the computer of the system or device read and execute the program. In addition, a circuit (e.g., ASI It can also be achieved by C).

[0108] The disclosure of this specification includes the following configurations, methods, and programs.

[0109] (Configuration 1) an image generation unit that generates a first image representing a field of view from a viewpoint of a first user in a virtual three-dimensional space shared by a plurality of users, and synthesizes an instruction UI (user interface) used by each user to point to a point in the three-dimensional space with the first image; a determination unit that determines visibility of a pointing position of the second user, which is a point in the three-dimensional space pointed to by the second user through the pointing UI, when viewed from a viewpoint of the first user; a UI display control unit that switches a display method of the second user's instruction UI in the first image according to a determination result by the determination unit; An information processing system having the above configuration.

[0110] (Configuration 2) the determination unit determines that the visibility of the pointing position of the second user is good when the pointing position of the second user is in a position visible from the viewpoint of the first user, and determines that the visibility of the pointing position of the second user is poor when the pointing position of the second user is in a position invisible from the viewpoint of the first user. 2. An information processing system according to configuration 1.

[0111] (Configuration 3) When the determination result by the determination unit is changed and the same determination result continues for a predetermined time, the UI display control unit switches a display method of the second user instruction UI in the first image. 3. The information processing system according to configuration 1 or 2.

[0112] (Configuration 4) the determination unit determines, using depth information of an object included in the first image, that the visibility of the pointing position of the second user is good when the pointing position of the second user is on the front side of the object, and determines that the visibility of the pointing position of the second user is poor when the pointing position of the second user is on the back side of the object. The information processing system according to any one of configurations 1 to 3.

[0113] (Configuration 5) the instruction UI includes a pointer representing an instruction position, which is a point in the three-dimensional space pointed to by a user, and a ray representing an instruction direction, which is a direction in which the user points; The UI display control unit, When the determination unit determines that the visibility of the pointing position of the second user is poor, a ray of the second user is displayed; When the determination unit determines that the visibility of the pointing position of the second user is good, the ray of the second user is hidden. The information processing system according to any one of configurations 1 to 4.

[0114] (Configuration 6) the instruction UI includes a pointer representing an instruction position, which is a point in the three-dimensional space pointed to by a user, and a ray representing an instruction direction, which is a direction in which the user points; The UI display control unit, When the determination unit determines that the visibility of the pointing position of the second user is poor, a method of displaying the pointer of the second user is changed; When the determination unit determines that the visibility of the pointing position of the second user is good, the ray of the second user is hidden. The information processing system according to any one of configurations 1 to 5.

[0115] (Configuration 7) the UI display control unit controls so as to display an instruction UI for the second user only when the first user is performing a predetermined operation; The information processing system according to any one of configurations 1 to 6.

[0116] (Configuration 8) the UI display control unit, when instruction UIs of a plurality of users overlap each other on the first image, changes a display method of the overlapping instruction UIs. The information processing system according to any one of configurations 1 to 7.

[0117] (Configuration 9) the determination unit determines that visibility of the pointing position of the second user is poor when a distance between a viewpoint of the first user and a pointing position of the second user is equal to or greater than a threshold, and determines that visibility of the pointing position of the second user is good when the distance is less than the threshold. The information processing system according to any one of configurations 1 to 8.

[0118] (Configuration 10) The threshold value is determined based on a distance between a viewpoint of the second user and a pointing position of the second user. 10. An information processing system according to configuration 9.

[0119] (Configuration 11) the determination unit determines that the visibility of the pointing position of the second user is good when an angle formed between the line of sight of the first user and a line connecting a viewpoint of the first user and a pointing position of the second user is within a predetermined angle range, and determines that the visibility of the pointing position of the second user is poor when the angle formed is outside the predetermined angle range. The information processing system according to any one of configurations 1 to 10.

[0120] (Configuration 12) The predetermined angle range is determined based on a viewing angle of the first user. 12. An information processing system according to configuration 11.

[0121] (Configuration 13) The determination unit is determining whether the viewpoint of the first user is on the front side or the back side of the object based on a surface normal of the object at the pointing position of the second user; determining that the pointing position of the second user is in a position visible from the viewpoint of the first user when the viewpoint of the first user is on the front side of the object and an angle formed between the line of sight of the first user and a line connecting the viewpoint of the first user and the pointing position of the second user is within a predetermined angle range; when the viewpoint of the first user is on the back side of the object and the angle is within the predetermined angle range, it is determined that the pointing position of the second user is within the field of view of the first user but is not visible from the viewpoint of the first user due to obscuration by the object; The information processing system according to any one of configurations 1 to 12.

[0122] (Configuration 14) the UI display control unit changes at least one of a size, a shape, and a transparency of the second user instruction UI according to a result of the determination by the determination unit. The information processing system according to any one of configurations 1 to 13.

[0123] (Configuration 15) The UI display control unit adds an annotation to the instruction UI of the second user according to a result of the determination by the determination unit. The information processing system according to any one of configurations 1 to 14.

[0124] (Configuration 16) the UI display control unit displays a message to inform the first user that the second user is giving an instruction through an instruction UI in response to a determination result by the determination unit. The information processing system according to any one of configurations 1 to 15.

[0125] (Configuration 17) the UI display control unit changes a display method of an object where a pointing position of the second user is located, according to a determination result by the determination unit. The information processing system according to any one of configurations 1 to 16.

[0126] (Configuration 18) The determination unit is Measure a time during which it is determined that the pointing position of the second user is visible from a viewpoint of the first user; changing a display method of the instruction UI according to the measured time; The information processing system according to any one of configurations 1 to 17.

[0127] (Method 19) 1. A computer-implemented information processing method, comprising: generating a first image representing a field of view from a viewpoint of a first user in a virtual three-dimensional space shared by a plurality of users, and synthesizing an instruction UI (user interface) used by each user to point to a point in the three-dimensional space onto the first image; determining visibility of a pointing position of the second user, which is a point in the three-dimensional space pointed to by the second user through the pointing UI, when viewed from a viewpoint of the first user; switching a display method of the second user instruction UI in the first image according to a result of the determination; An information processing method comprising the steps of:

[0128] (Program 20) A program for causing a computer to execute each step of the information processing method described in Method 19. [Explanation of symbols]

[0129] 1: Information processing system 106: UI display control unit 109: Image generation unit 112: Judgment section

Claims

1. An image generation unit generates a first image representing the field of view from the viewpoint of a first user in a virtual three-dimensional space shared by multiple users, and synthesizes an instruction UI (user interface) used by each user to point to a point in the three-dimensional space onto the first image. A determination unit that determines the visibility of the point in the three-dimensional space indicated by the second user via the instruction UI, when viewed from the viewpoint of the first user, A UI display control unit switches the display method of the second user instruction UI in the first image according to the determination result of the determination unit, An information processing system having

2. The determination unit determines that the visibility of the second user's indicated position is good if the indicated position is visible from the first user's viewpoint, and determines that the visibility of the second user's indicated position is poor if the indicated position is not visible from the first user's viewpoint. The information processing system according to claim 1.

3. After the determination result by the determination unit changes, if the same determination result persists for a predetermined period of time, the UI display control unit switches the display method of the second user instruction UI in the first image. The information processing system according to claim 1.

4. The determination unit uses the depth information of the object included in the first image to determine that the visibility of the second user's indicated position is good when the indicated position is in front of the object, and determines that the visibility of the second user's indicated position is poor when the indicated position is behind the object. The information processing system according to claim 1.

5. The instruction UI includes a pointer representing an instruction position, which is a point in the three-dimensional space pointed to by the user, and a ray representing an instruction direction, which is the direction pointed to by the user. The UI display control unit is If the determination unit determines that the visibility of the second user's indicated position is poor, the second user's ray is displayed. If the determination unit determines that the visibility of the second user's indicated position is good, the second user's ray is hidden. The information processing system according to claim 1.

6. The instruction UI includes a pointer representing an instruction position, which is a point in the three-dimensional space pointed to by the user, and a ray representing an instruction direction, which is the direction pointed to by the user. The UI display control unit is If the determination unit determines that the visibility of the second user's indicated position is poor, the display method of the second user's pointer is changed. If the determination unit determines that the visibility of the second user's indicated position is good, the second user's ray is hidden. The information processing system according to claim 1.

7. The UI display control unit controls the display of the second user's instruction UI only when the first user is performing a predetermined operation. The information processing system according to claim 1.

8. The UI display control unit changes the display method of the overlapping instruction UIs when multiple user instruction UIs overlap each other on the first image. The information processing system according to claim 1.

9. The determination unit determines that the visibility of the second user's indicated position is poor if the distance between the first user's viewpoint and the second user's indicated position is greater than or equal to a threshold, and determines that the visibility of the second user's indicated position is good if the distance is less than the threshold. The information processing system according to claim 1.

10. The threshold is determined based on the distance between the viewpoint of the second user and the position indicated by the second user. The information processing system according to claim 9.

11. The determination unit determines that the visibility of the second user's indicated position is good if the angle between the first user's line of sight and the line connecting the first user's viewpoint and the second user's indicated position is within a predetermined angular range, and determines that the visibility of the second user's indicated position is poor if the angle is outside the predetermined angular range. The information processing system according to claim 1.

12. The predetermined angle range is determined based on the field of view of the first user. The information processing system according to claim 11.

13. The determination unit, Based on the surface normal of the object at the position indicated by the second user, it is determined whether the first user's viewpoint is on the front or back side of the object. If the viewpoint of the first user is on the front side of the object, and the angle between the line of sight of the first user and the line connecting the viewpoint of the first user and the indicated position of the second user is within a predetermined angular range, then it is determined that the indicated position of the second user is visible from the viewpoint of the first user. The viewpoint of the first user is on the back side of the object, and the angle it makes is the predetermined angle. If it is within a certain range, it is determined that the second user's indicated position is within the first user's field of view, but is not visible from the first user's viewpoint due to obstruction by the object. The information processing system according to claim 1.

14. The UI display control unit changes at least one of the size, shape, and transparency of the second user instruction UI according to the determination result by the determination unit. The information processing system according to claim 1.

15. The UI display control unit adds annotations to the second user instruction UI according to the determination result by the determination unit. The information processing system according to claim 1.

16. The UI display control unit displays a message to inform the first user that the second user is giving instructions via the instruction UI, in accordance with the determination result by the determination unit. The information processing system according to claim 1.

17. The UI display control unit changes the display method of the object where the second user's indicated position is located, according to the determination result by the determination unit. The information processing system according to claim 1.

18. The determination unit, The time during which the location indicated by the second user is deemed to be visible from the first user's perspective is measured. The UI display control unit is The method of displaying the instruction UI is changed according to the measured time. The information processing system according to claim 1.

19. A method of information processing performed by a computer, The steps include generating a first image representing the field of view from the viewpoint of a first user for a virtual three-dimensional space shared by multiple users, and compositing an instruction UI (user interface) used by each user to point to a point in the three-dimensional space onto the first image, A step of determining the visibility of the point in the three-dimensional space indicated by the second user via the instruction UI, which is the position indicated by the second user, when viewed from the viewpoint of the first user; Depending on the result of the determination, the step of switching the display method of the second user instruction UI in the first image, An information processing method having

20. A program for causing a computer to perform each step of the information processing method described in claim 19.

Images