Information processing apparatus, information processing method, and storage medium

Abstract

The present disclosure relates to an information processing apparatus, an information processing method, and a storage medium. The information processing apparatus includes an instruction acquisition unit configured to acquire, from a user who is one of a plurality of users in a mixed reality space, an instruction for a virtual object in the mixed reality space; a mode determination unit configured to determine, based on the acquired instruction, a display mode of the virtual object for other users in the mixed reality space; a viewpoint determination unit configured to determine, based on the determined display mode, a viewpoint for the virtual object in the mixed reality space; and an image generation unit configured to generate an image of the mixed reality space including the virtual object viewed from the determined viewpoint.

Description

Technical Field

[0001] This disclosure relates to information processing techniques for overlaying images of virtual objects onto images in real space. Background Technology

[0002] In recent years, mixed reality (MR) technology has become increasingly popular. MR technology can provide mixed reality to users wearing head-mounted displays (HMDs) by displaying overlaid images, in which computer-generated virtual images are superimposed on images of the real space captured by the HMD's cameras.

[0003] Japanese Patent Application Publication No. 2017-120650 discusses a technique for presenting a virtual object to users at multiple locations in the direction in which the virtual object is displayed in a virtual space as viewed by other users. Japanese Patent Application Publication No. 2020-74066 discusses a technique for enabling users to share MR spatial video images viewed by other users and further manipulate the MR spatial video images on the user's own terminal.

[0004] When multiple users are simultaneously viewing virtual objects arranged in MR space, users may need to move in the real space, or users other than a specific user may need to move their viewpoint position within that specific user's virtual space. This is impractical. Summary of the Invention

[0005] This disclosure relates to techniques for enabling multiple users to view virtual objects without moving their location in real space or their viewpoint location in virtual space.

[0006] According to one aspect of this disclosure, an information processing apparatus includes: an instruction acquisition unit configured to acquire an instruction for a virtual object in a mixed reality space from a user who is one of a plurality of users in the mixed reality space; a mode determination unit configured to determine a display mode of the virtual object for other users in the mixed reality space based on the acquired instruction; a viewpoint determination unit configured to determine a viewpoint of the virtual object in the mixed reality space based on the determined display mode; and an image generation unit configured to generate an image of the mixed reality space including the virtual object as viewed from the determined viewpoint.

[0007] Other features of this disclosure will become apparent from the following description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0008] Figure 1 This is a block diagram illustrating an example of the hardware configuration of an information processing device according to a first exemplary embodiment.

[0009] Figure 2 This is a functional block diagram illustrating the functional configuration of an information processing device according to a first exemplary embodiment.

[0010] Figure 3 This is a diagram illustrating an example of a mixed reality space according to a first exemplary embodiment.

[0011] Figure 4 This is a diagram illustrating an example of a presenter viewpoint sharing mode according to a first exemplary embodiment.

[0012] Figure 5 This is a diagram illustrating an example of the optimal viewpoint sharing mode according to a first exemplary embodiment.

[0013] Figure 6 This is a flowchart illustrating the information processing procedure according to a first exemplary embodiment.

[0014] Figure 7 This is a flowchart illustrating the viewpoint determination process and image generation process according to a first exemplary embodiment.

[0015] Figure 8A and Figure 8B This is a diagram illustrating an example of a mixed reality space according to a second exemplary embodiment.

[0016] Figure 9 This is a diagram illustrating an example of the arrangement of virtual avatars according to a second exemplary embodiment.

[0017] Figure 10 This is a flowchart illustrating the viewpoint determination process and image generation process according to a second exemplary embodiment. Detailed Implementation

[0018] Exemplary embodiments of the present disclosure will now be described with reference to the accompanying drawings. The exemplary embodiments described below are not intended to limit the present disclosure, and not all combinations of features described in the exemplary embodiments are necessary for the solutions provided by the present disclosure. Configurations according to the exemplary embodiments may be appropriately modified or changed depending on the specifications and conditions of the device to which any exemplary embodiment is applied (e.g., usage conditions and usage environment). Parts of the exemplary embodiments described below may be appropriately combined with each other. In the exemplary embodiments described below, the same components are indicated by the same reference numerals.

[0019] Figure 1 An example of the hardware configuration of the information processing device 201 according to a first exemplary embodiment is shown; Figure 2 This is a functional block diagram illustrating the functional configuration of the information processing device 201 according to this exemplary embodiment. In the description... Figure 1 and Figure 2 Before the configuration shown, please refer to Figures 3 to 5 This describes a scenario where multiple users simultaneously view virtual objects arranged in a mixed reality (MR) space using head-mounted displays (HMDs).

[0020] As an example of this situation, a scenario will be described where a specific user is the presenter and explains a car to participating users by displaying overlaid images on the user's various HMDs, in which images representing virtual objects of the car are superimposed on images of real space. Participants are users other than the specific user. During a car demonstration, the specific user, as the presenter, may sometimes change the area or direction the presenter wants the participating users to focus on. In this case, using existing MR technology, the user would move in real space, or the participant would move their viewpoint position in the presenter's virtual space, which is impractical. To address this problem, the information processing device 201 according to this exemplary embodiment performs the information processing described below.

[0021] According to this exemplary embodiment, the information processing device 201 can change the viewpoint of other users toward a virtual object in response to an instruction from a specific user toward a virtual object. In this exemplary embodiment, the viewpoint toward a virtual object includes the position and orientation of the viewpoint. Therefore, according to this exemplary embodiment, the information processing device 201 enables multiple users to view a virtual object without moving their position in real space or their viewpoint position in virtual space. Furthermore, according to this exemplary embodiment, the information processing device 201 executes a first display mode and a second display mode, wherein in the first display mode, the position and orientation of the specific user's viewpoint are shared among other users, and in the second display mode, the optimal position and orientation of a pre-set viewpoint toward the virtual object are shared among users. In this exemplary embodiment, the first display mode, in which the position and orientation of the specific user's viewpoint are shared among other users, is called a "presenter viewpoint sharing mode." The second display mode, in which the optimal position and orientation of a pre-set viewpoint toward the virtual object are shared among users, is called an "optimal viewpoint sharing mode." In this exemplary embodiment, in the optimal viewpoint sharing mode, the optimal position and orientation of a pre-set viewpoint toward the virtual object can be shared among users other than the specific user, or among all users including the specific user.

[0022] When either the presenter viewpoint sharing mode or the optimal viewpoint sharing mode is set, the information processing device 201 according to this exemplary embodiment enables a user to share the viewpoint position and orientation of a virtual object based on the set display mode. The display mode can be preset to use either the presenter viewpoint sharing mode or the optimal viewpoint sharing mode, or if not preset, the settings associated with the virtual object to be displayed can be followed. Optionally, the display mode to be used can be determined, for example, by the operation of a mouse or controller by a specific user or other users. In other words, the display mode to be used can be optionally determined for each user.

[0023] Figure 3 An example is shown where multiple users 101 to 104 (specific user 101 and other users 102 to 104) are in a real space 100, and a virtual object 105 is arranged in a virtual space corresponding to the real space 100. The virtual object 105 represents a typical car including two headlights 106. Each user 101 to 104 wears an HMD equipped with a camera device (camera). It is assumed that the HMDs of each user 101 to 104 display an overlay image in which the virtual object 105 is superimposed on an image of the real space 100, which is a captured image of the real space 100. In other words, each user 101 to 104 uses an HMD to view the virtual object 105, which is virtually arranged in the real space 100. Figure 3 Images 107 to 110, displayed on the HMDs of users 101 to 104, are also shown. Image 107 represents the image displayed on the HMD of user 101. Similarly, image 108 represents the image displayed on the HMD of user 102, image 109 represents the image displayed on the HMD of user 103, and image 110 represents the image displayed on the HMD of user 104. In this example, user 101 is the specific user acting as the presenter, and the other users 102 to 104 are the participants in the presentation.

[0024] Suppose user 101 (the presenter) indicates that the headlight 106 of the virtual object 105 is the area the presenter wants users 102 to 104 (participants) to focus on. In this case, the information processing device 201 enables users 101 to 104 to share the viewpoint position and orientation of the virtual object 105 based on the set display mode (i.e., presenter viewpoint sharing mode or optimal viewpoint sharing mode).

[0025] For example, in the presenter viewpoint sharing mode, the information processing device 201 acquires the viewpoint position and orientation of user 101 (the presenter) regarding a virtual object 105 in virtual space. The information processing device 201 then overlays the virtual object 105 viewed from user 101's viewpoint position and orientation onto images of the real space 100 captured by cameras on the HMDs of users 102 to 104 (participants), respectively, to generate an overlaid image, which is then displayed on the respective HMDs of users 102 to 104. Therefore, each HMD of users 102 to 104 displays an image of the headlight 106 of the virtual object 105 viewed from the viewpoint position and orientation of user 101 (the specific user).

[0026] For example, in the optimal viewpoint sharing mode, the information processing device 201 acquires the optimal viewpoint position and direction, which are pre-set to be the most visible viewpoint position and direction for the headlight 106 of the virtual object 105. Then, the information processing device 201 overlays the virtual object 105 viewed from the optimal viewpoint position and direction onto images of the real space 100 captured by cameras on the HMDs of users 102 to 104, respectively, to generate an overlaid image, which is then displayed on the respective HMDs of users 102 to 104. Therefore, each HMD of users 102 to 104 displays an image of the headlight 106 of the virtual object 105 viewed from the optimal viewpoint position and direction. In the optimal viewpoint sharing mode, the optimal viewpoint position and direction for the virtual object 105 does not necessarily need to be shared with a specific user 101; however, in this exemplary embodiment, the optimal viewpoint position and direction are also shared with the HMD of a specific user 101. As described above, the information processing device 201 enables users 101 to 104 to share the viewpoint position and direction for the virtual object 105 based on the set display mode.

[0027] Figure 4 An example of a presenter viewpoint sharing mode is illustrated. Images 601 to 604 represent images displayed on the HMDs of users 101 to 104, respectively. In presenter viewpoint sharing mode, if a specific user 101 points to the headlight 106 of a virtual object 105, the information processing device 201 shares the viewpoint position and orientation of user 101 toward the virtual object 105 in virtual space among the HMDs of other users 102 to 104. Therefore, each user's HMD displays an overlay image in which the virtual object 105, viewed from the viewpoint position and orientation of the specific user 101 (presenter), is superimposed on an image of real space 100 viewed from the viewpoint position and orientation of the corresponding user in real space 100.

[0028] Figure 5The optimal viewpoint sharing mode is illustrated. Images 701 to 704 represent images displayed on the HMDs of users 101 to 104, respectively. The optimal viewpoint position 705 indicates the most visible viewpoint position and direction of the headlights 106 of the virtual object 105. When a specific user 101 points the headlights 106 of the virtual object 105 in the optimal viewpoint sharing mode, the information processing device 201 shares the optimal viewpoint position 705 for the headlights 106 among the HMDs of users 101 to 104 in the real space 100.

[0029] Therefore, each HMD of users 101 to 104 displays an overlay image in which a virtual object 105 seen from the best viewpoint position 705 and best direction view of the headlight 106 is superimposed on an image of the real space 100 seen from the viewpoint position of the corresponding user in the real space 100.

[0030] The configuration and operation of the information processing device 201 according to this exemplary embodiment will now be described.

[0031] Figure 1 A personal computer (PC) configuration is shown as an example of the hardware configuration of the information processing device 201. The information processing device 201 overlays virtual objects 105 generated using computer graphics (CG) technology onto images of the real space 100 captured by camera devices (cameras) installed on the HMDs of users 101 to 104 to generate MR spatial images, and sends the MR spatial images to the respective HMDs. MR technology is prior art, therefore its detailed description is omitted.

[0032] The central processing unit (CPU) 202 is the system control unit and controls the information processing device 201 as a whole. The CPU 202 executes information processing programs to realize information processing according to this exemplary embodiment.

[0033] Read-only memory (ROM) 203 stores unchanging programs and parameters (such as basic programs and initial data). Random access memory (RAM) 204 temporarily stores input information and calculation results for information processing and image processing.

[0034] The operation unit 210 includes operating devices such as a keyboard, mouse, and controller, which can be used to perform pointing operations or input various commands, and acquires operation instructions and commands from any of the users 101 to 104. In this exemplary embodiment, examples of information from the operation unit 210 include instruction information acquired when a specific user 101 inputs an instruction for the virtual object 105. The instruction information acquired when a specific user 101 inputs an instruction for the virtual object 105 is sent to the input detection unit 206. The operation unit 210 may not only have the function of acquiring instructions input using a pointing device, but may also have the function of acquiring instructions input via voice.

[0035] The input detection unit 206 receives the input of the required data. In this exemplary embodiment, examples of the required data include operation instructions and commands input via the operation unit 210.

[0036] Storage unit 205 is a device capable of reading and writing various types of information (such as a hard disk or memory card built into or externally attached to information processing device 201, or a memory card, floppy disk, or integrated circuit (IC) card that can be attached to and removed from information processing device 201). According to this exemplary embodiment, an information processing program is stored in storage unit 205. The information processing program is read from storage unit 205, loaded into RAM 204, and executed by CPU 202. The information processing program may be stored in ROM 203. Storage unit 205 may also store information related to operation instructions input by a specific user 101 via operation unit 210 and input detection unit 206, position and posture information related to users 101 to 104 detected by each HMD, and position and posture information related to virtual object 105. Examples of instructions input by a specific user 101 include instructions for areas of the virtual object 105 that the specific user 101 wants other users 102 to 104 to focus on, and instructions for any part into which the virtual object 105 is divided. In this exemplary embodiment, the storage unit 205 also stores information related to the optimal viewpoint position and orientation preset for the virtual object 105.

[0037] The graphics processing unit (GPU) board 208 is a general-purpose graphics board and performs processes such as image generation and compositing. According to this exemplary embodiment, the GPU board 208 can perform image generation and compositing by overlaying an image of a virtual object 105 generated using CG technology onto an image of the real space 100 captured by the camera of the HMD. The image of the virtual object 105 can be pre-generated and stored in the storage unit 205.

[0038] Display driver 209 is software used to control display unit 211.

[0039] Display unit 211 is an electronic display device, such as a liquid crystal display, installed on each of the HMDs of users 101 to 104. According to this exemplary embodiment, display unit 211 enables each of users 101 to 104 to input commands using, for example, a mouse cursor (not shown) on a display, and also displays images such as MR spatial images generated by GPU board 208 under the control of CPU 202 based on information in storage unit 205. Figure 1 The example assumes processing using a PC, therefore the operation unit 210 and display unit 211 are externally connected to the information processing device 201. In the case where the HMD has information processing functions according to this exemplary embodiment, the operation unit 210 and display unit 211 are included in the information processing device 201.

[0040] The communication interface (I / F) 207 is an interface unit capable of sending data to or receiving data from an operating device or cloud. In this exemplary embodiment, the communication I / F 207 can receive data from images of the real space 100 captured by cameras mounted on the HMDs of users 101 to 104, as well as position and posture information detected by the HMDs of users 101 to 104, via a network.

[0041] Figure 2 This is a functional block diagram illustrating an example of the functional configuration of an information processing device 201 according to this exemplary embodiment.

[0042] The information processing device 201 includes an operation unit 210 as a functional module, an object instruction acquisition unit 305, a mode determination unit 306, a viewpoint determination unit 307, an image generation unit 308, a user position and posture storage unit 301, and an object position and posture storage unit 303, wherein the user position and posture storage unit 301 and the object position and posture storage unit 303 correspond to the position and posture acquisition unit.

[0043] User position and posture storage unit 301 acquires and stores the positions and postures of each user 101 to 104 in the mixed reality space (MR space). In this exemplary embodiment, user position and posture storage unit 301 acquires information related to position coordinates and gaze vectors from each HMD worn by users 101 to 104, and stores the acquired information as position and posture information related to each user 101 to 104 (referred to as user position and posture information 302). User position and posture information 302 is stored, for example, in storage unit 205.

[0044] The object position and pose storage unit 303 acquires and stores the position and pose of the virtual object 105 in the mixed reality space. In this exemplary embodiment, the object position and pose storage unit 303 acquires information such as the position, orientation, and size of the virtual object 105 in the virtual space, and stores the acquired information as object position and pose information 304. The object position and pose information 304 is stored, for example, in the storage unit 205.

[0045] The operation unit 210 obtains various instructions and commands (such as virtual object selection instructions and display mode switching instructions) from any one of the users 101 to 104.

[0046] The object instruction acquisition unit 305 acquires information related to user instructions for the virtual object 105 input via the operation unit 210 in the mixed reality space. In this exemplary embodiment, the object instruction acquisition unit 305 acquires instructions for the virtual object 105 input by a specific user 101, but it may also acquire instructions input by any of the other users 102 to 104.

[0047] The mode determination unit 306 determines whether to use the presenter viewpoint sharing mode or the optimal viewpoint sharing mode as the display mode. If a display mode to be used is preset, the mode determination unit 306 determines the display mode based on that setting. If any of the users 101 to 104 inputs an instruction related to which display mode to use via the operation unit 210, the mode determination unit 306 determines the display mode based on the input instruction. If no display mode is preset or if any of the users 101 to 104 does not input an instruction related to which display mode to use, the mode determination unit 306 determines the display mode based on settings associated with the virtual object 105 to be displayed. In this exemplary embodiment, a specific user 101 (presenter) inputs an instruction related to which display mode to use, but any of the other users 102 to 104 (participants) can input the instruction.

[0048] The viewpoint determination unit 307 determines the viewpoint for the virtual object 105 in the mixed reality space based on the display mode determined by the mode determination unit 306. In this exemplary embodiment, the viewpoint determination unit 307 determines the viewpoint position and direction (viewing direction) for the virtual object 105 based on the user position and posture information 302, the object position and posture information 304, and the display mode determined by the mode determination unit 306. For example, in the case of a presenter viewpoint sharing mode, the viewpoint determination unit 307 determines the viewpoint position and direction of a specific user 101 for the virtual object 105 in the virtual space as the viewpoint position and direction to be shared among the HMDs of other users 102 to 104. In the case of an optimal viewpoint sharing mode, the viewpoint determination unit 307 determines the optimal viewpoint position and direction for the pre-set headlight 106 of the virtual object 105 to be most visible as the viewpoint position and direction to be shared among the HMDs of users 101 to 104. The viewpoint determination unit 307 can optionally change the user's shared viewpoint position and orientation based on instructions input, for example, via the operation unit 210.

[0049] Image generation unit 308 generates an image of a mixed reality space including virtual object 105 as viewed from the viewpoint determined by viewpoint determination unit 307. In this exemplary embodiment, image generation unit 308 overlays an image of virtual object 105 onto images of real space 100 captured by cameras of users 101 to 104's HMDs, based on user position and posture information 302, object position and posture information 304, and viewpoint position and orientation determined by viewpoint determination unit 307, to generate an overlaid image. Then, image generation unit 308 sends the image of real space 100 overlaid with virtual object 105 to the respective HMDs of users 101 to 104. Therefore, each user's HMD displays an overlaid image of real space 100 with virtual object 105 overlaid on it. More specifically, in the presenter viewpoint sharing mode, each of the HMDs of other users 102 to 104 displays an overlay image in which the virtual object 105, viewed from the viewpoint and direction of a specific user 101, is superimposed on an image of the real space 100 captured by the camera of the respective user's HMD. In the optimal viewpoint sharing mode, each of the HMDs of users 101 to 104 displays an overlay image in which the virtual object 105, viewed from a preset optimal viewpoint and direction, is superimposed on an image of the real space 100 captured by the camera of the respective user's HMD.

[0050] Figure 6 This is a flowchart illustrating the information processing process performed by the information processing device 201.

[0051] The processing in the flowchart is implemented by executing the information processing program according to this exemplary embodiment via CPU 202.

[0052] In step S401, the user position and pose storage unit 301 uses existing self-position estimation techniques to analyze the real-space images acquired by the cameras (video recording devices) mounted on the HMDs of each user 101 to 104, and acquires position and pose information related to each user 101 to 104. In this exemplary embodiment, the position and pose information related to each user 101 to 104 is the position and pose of the HMD, and the position and pose of the HMD is information that can represent the viewpoint position and orientation of the corresponding user. Examples of self-position estimation methods include Simultaneous Localization and Mapping (SLAM). The method is not limited to SLAM, as long as the method can acquire the position and pose (HMD position and pose) of each user 101 to 104 in the real space 100. For example, a Global Positioning System (GPS), a gyroscope sensor, or a geomagnetic sensor can be used to acquire the position and pose of each user 101 to 104. The user position and pose storage unit 301 stores the information acquired in step S401 as user position and pose information 302.

[0053] In step S402, the user position and posture storage unit 301 determines whether position and posture information related to all users 101 to 104 in the mixed reality space has been acquired. If the acquisition of position and posture information (viewpoint position and orientation of all users 101 to 104) related to all users 101 to 104 has not been completed ("No" in step S402), the process returns to step S401 to acquire position and posture information related to any user 101 to 104 that has not yet been acquired. On the other hand, if the acquisition is completed ("Yes" in step S402), the process proceeds to step S403.

[0054] In step S403, the object position and pose storage unit 303 acquires position and pose information (viewpoint position and orientation of the virtual object 105) related to the virtual object 105. In this exemplary embodiment, since the specific user 101 is the presenter, the storage unit 205 stores an overlay image of the virtual object 105 superimposed on an image of the real space 100 captured by the presenter's HMD camera. The object position and pose storage unit 303 loads the overlay image from the storage unit 205 into the RAM 204 and acquires position and pose information related to the virtual object 105 selected by the presenter via the operation unit 210 in the virtual space from the overlay image. Then, the object position and pose storage unit 303 stores the acquired information as object position and pose information 304.

[0055] In step S404, the object instruction acquisition unit 305 acquires instructions for the virtual object 105 from a specific user 101 (the presenter) based on the operation input via the operation unit 210. The instructions from the presenter may not be input via the operation unit 210, but rather via voice input.

[0056] In step S405, the mode determination unit 306 determines whether to use the presenter viewpoint sharing mode or the best viewpoint sharing mode as the display mode for the virtual object 105 that has received a notification from the object instruction acquisition unit 305.

[0057] In step S406, the viewpoint determination unit 307 determines the viewpoint position and orientation of each user 101 to 104 relative to the virtual object 105 based on the user position and posture information 302, the object position and posture information 304, and the display mode determined by the mode determination unit 306. (Refer to the following...) Figure 7 The flowchart shown describes in detail the display viewpoint determination process of the viewpoint determination unit 307.

[0058] In step S407, the image generation unit 308, based on the user position and pose information 302, the object position and pose information 304, and the viewpoint position and direction determined by the viewpoint determination unit 307, overlays the virtual object 105 onto the corresponding real-space image to generate an overlaid image. (Refer to the following...) Figure 7 The flowchart shown describes in detail the superimposed image generation process of the image generation unit 308.

[0059] In step S408, the display unit 211 displays an overlay image on the HMD of each user 101 to 104, in which the virtual object 105 is overlaid on the corresponding real space image.

[0060] Figure 7 This is a flowchart illustrating the display viewpoint determination process performed by the viewpoint determination unit 307 in step S406 and the overlay image generation process performed by the image generation unit 308 in step S407. The processes in steps S501 to S506 correspond to the processes in step S406, and the processes in steps S507 and S508 correspond to the processes in step S407.

[0061] In step S501, the viewpoint determination unit 307 determines whether the display mode determined in step S405 is the presenter viewpoint sharing mode or the optimal viewpoint sharing mode. If the viewpoint determination unit 307 determines that the display mode is the presenter viewpoint sharing mode ("Yes" in step S501), the process proceeds to step S502. If the viewpoint determination unit 307 determines that the display mode is the optimal viewpoint sharing mode ("No" in step S501), the process proceeds to step S503.

[0062] In step S502, the viewpoint determination unit 307 obtains the viewpoint position and orientation of a specific user 101 (presenter) based on the user position and posture information 302 and the object position and posture information 304. Then, the process proceeds to step S504.

[0063] In step S504, the viewpoint determination unit 307 determines whether the gaze of a specific user 101 (the presenter) has left the target virtual object (virtual object 105). If the viewpoint determination unit 307 determines that the presenter's gaze has not left the target virtual object ("No" in step S504), the process proceeds to step S506. In step S506, the viewpoint determination unit 307 shares the acquired viewpoint position and direction among the HMDs of other users 102 to 104.

[0064] On the other hand, if the viewpoint determination unit 307 determines that the presenter's gaze has left the target virtual object ("Yes" in step S504), the process proceeds to step S505. More specifically, for example, when the presenter is explaining the headlight 106 of the virtual object 105 while using a pointing device to indicate the headlight 106, the presenter sometimes looks away from the virtual object 105 to read explanatory notes. If the presenter's viewpoint position and direction are shared among other users 102 to 104 at this time, the gaze of the other users 102 to 104 also leaves the virtual object 105. In other words, the display on the HMD of the other users 102 to 104 also leaves the virtual object 105. Therefore, if the viewpoint determination unit 307 determines that the presenter's gaze has left the target virtual object ("Yes" in step S504), the process proceeds to step S505.

[0065] In step S505, the viewpoint determination unit 307 acquires the viewpoint position and direction of the presenter when looking at the virtual object 105, and the process proceeds to step S506. More specifically, in this case, in step S506, the viewpoint determination unit 307 shares the presenter's viewpoint position and direction when looking at the virtual object 105 among the HMDs of other users 102 to 104. Alternatively, in step S506, the viewpoint determination unit 307 may share a pre-set viewpoint position and direction for the virtual object 105 among the HMDs of other users 102 to 104.

[0066] On the other hand, when the process proceeds from step S501 to step S503, the viewpoint determination unit 307 obtains the optimal viewpoint position and direction preset for the virtual object 105, and the process continues to step S506. In step S506, the viewpoint determination unit 307 shares the optimal viewpoint position and direction among the HMDs of other users 102 to 104.

[0067] Following the processing in step S506, the process proceeds to step S507, where the image generation unit 308 performs the processing. In step S507, the image generation unit 308 generates a superimposed image based on the viewpoint position and orientation shared among the other users 102 to 104 in step S506. The superimposed image can be generated on the corresponding HMD, or a superimposed image generated by other components can be sent to the corresponding HMD.

[0068] In step S508, the information processing device 201 determines whether the specific user 101 (the presenter) has ended the presentation. If the presentation has ended (yes in step S508), the image generation unit 308 sends an image or message indicating the end of the presentation to the display unit 211. On the other hand, if the presentation has not ended (no in step S508), the information processing device 201 repeats the processing in steps S501 to S507 until the presentation ends. This process is repeated during the presentation. Figure 7 While processing steps S501 to S507, the display mode can be optionally switched.

[0069] In the first exemplary embodiment described above, an overlay image is generated by sharing the viewpoint position and orientation of a specific user 101 relative to the virtual object 105, or by a pre-set optimal viewpoint position and orientation for the virtual object 105, and then the overlay image is displayed on the respective HMDs of other users 102 to 104. For example, in the case where multiple users 101 to 104 surround and view the virtual object 105 in MR space, users 101 to 104 may want to discuss the virtual object 105 while viewing it from different positions relative to each other. Therefore, in the second exemplary embodiment, an example will be described in which different viewpoint positions and orientations are calculated for each user 102 to 104 relative to the virtual object 105, and an overlay image generated based on the different viewpoint positions and orientations is displayed on the respective HMDs of users 102 to 104.

[0070] Figure 8A and Figure 8B An example is shown where different viewpoint positions and orientations are calculated for other users 102 to 104 using the viewpoint position and orientation of a specific user 101 (the presenter) as a baseline.

[0071] Figure 8A The viewpoint positions and orientations of users 801 to 804 (specific user 802 and other users 801, 803, and 804) in virtual space 800 are shown. Figure 8AIn this example, user 802 corresponds to a specific user, and the viewpoint position and orientation of user 802 relative to the virtual object 105 in the virtual space 800 are used as a reference. In the virtual space 800, users 801 to 804 are arranged in a predetermined arrangement, such as side-by-side or fan-shaped, and the viewpoint positions and orientations of other users 801, 803, and 804 arranged in the predetermined arrangement are calculated based on the viewpoint position and orientation of the specific user 802 used as a reference. In this exemplary embodiment, although the viewpoint position and orientation of the specific user 802 are used as a reference, the viewpoint position and orientation used as a reference can be the viewpoint position and orientation of any of the selected users 801 to 804, or a viewpoint position and orientation preset for the virtual object 105.

[0072] Figure 8B This illustration depicts users 101 to 104 in a real space 100 and a virtual object 105 in a virtual space within a mixed reality space, according to this exemplary embodiment. Figure 8B Images 805 to 808 shown represent overlay images to be displayed on the respective HMDs of users 101 to 104.

[0073] In this exemplary embodiment, each of the HMDs for users 101 to 104 displays an overlay image, in which... Figure 8A The virtual object 105 viewed by one of the users 801 to 804, positioned and oriented in the virtual space 800, is superimposed on an image of the real space 100 captured by the camera of the HMD. More specifically, in this exemplary embodiment, users 101 to 104 can discuss the virtual object 105 as if viewing it from different locations (such as their positions in the virtual space 800) while maintaining their standing positions in the real space 100. Furthermore, in this exemplary embodiment, each user 101 to 104 can independently perform various operations (such as zooming in, zooming out, and rotating) on ​​the superimposed image displayed on the corresponding HMD via the operation unit 210.

[0074] Figure 9 This illustrates the method according to this exemplary embodiment. Figure 8A The illustration shows an example of virtual avatars 902, 903, and 904 of various users (e.g., users 801, 802, and 803) in virtual space 800 being displayed simultaneously overlaid on a real-world image. Image 900 represents, for example, an image displayed on the HMD of user 104.

[0075] exist Figure 9In the example, the image generation unit 308 generates image 900 based on the viewpoint positions and orientations of users 801 to 804 in virtual space 800 by arranging and overlaying the virtual avatars 902 to 904 of other users 101 to 103 onto the image to be displayed on user 104's HMD. Therefore, even if users 101 to 104 are far apart from each other in real space 100, users 101 to 104 can feel as if they are close to each other and discussing virtual object 105 in virtual space.

[0076] Such a virtual avatar can be superimposed on an image generated according to the first exemplary embodiment, corresponding to the viewpoint determined by the viewpoint determination unit 307 for each user 101 to 104. In this case, the virtual avatar is generated by the image generation unit 308 according to the first exemplary embodiment.

[0077] Figure 10 This is a flowchart illustrating the process according to this exemplary embodiment. Figure 6 The process of display viewpoint determination by viewpoint determination unit 307 in step S406 and overlay image generation by image generation unit 308 in step S407. In steps S1001 to S1003, similar to... Figure 7 The processes in steps S501 to S503 include determining whether the presenter's viewpoint position and orientation are shared among other users 102 to 104 and obtaining the viewpoint position and orientation based on the determination result.

[0078] In this exemplary embodiment, in step S1002, the viewpoint determination unit 307 obtains the viewpoint position and orientation of the presenter (specific user 101) from the user position and posture information 302 and the object position and posture information 304, and then the process proceeds to step S1004.

[0079] In step S1004, the viewpoint determination unit 307 calculates the viewpoint position and direction of each user 102 to 104 in the virtual space based on the viewpoint position and direction obtained in step S1002 or S1003. In this exemplary embodiment, assuming that users 101 to 104 are arranged in a predetermined arrangement (such as side-by-side or fan-shaped arrangement) relative to the target virtual object (virtual object 105), the viewpoint determination unit 307 calculates the viewpoint position and direction of each user 102 to 104. Therefore, each user 102 to 104 can view the virtual object 105 from different angles.

[0080] After the processing in step S1004, in step S1005, the viewpoint determination unit 307 uses the HMD of the corresponding user to share the viewpoint position and direction of each user 102 to 104 for the virtual object 105 in the virtual space, which was calculated in step S1004.

[0081] In step S1006, similar to the processing in step S507 according to the first exemplary embodiment, the image generation unit 308 generates an overlay image based on the viewpoint position and orientation shared with each user 102 to 104. The overlay image of the virtual object 105 superimposed on the real-space image can be generated on the HMD, or it can be generated by other components and sent to the HMD. In the case of generating an overlay image on the HMDs of each user 102 to 104, as referenced above... Figure 9 The virtual avatars can be generated based on the viewpoints of other users in the virtual space, and then simultaneously placed within the field of view. Therefore, discussions between users 101 to 104 can feel more immersive.

[0082] Following the processing in step S1006, in step S1007, similar to the processing in step S508, the information processing device 201 determines whether the presentation has ended. If the presentation has ended (yes in step S1007), the image generation unit 308 sends a notification that the presentation has ended to the display unit 211. If the presentation has not ended (no in step S1007), the information processing device 201 repeats the processing in steps S1001 to S1006 until the presentation ends. Furthermore, in this exemplary embodiment, the display mode may be optionally switched during the repetition of the processing in steps S1001 to S1007.

[0083] As described above, the information processing devices 201 of the first and second exemplary embodiments enable other users 102 to 104 to change and share the viewpoint position and orientation of the virtual object 105 in response to instructions from a specific user 101 (such as a presenter). Furthermore, the information processing devices 201 of the first and second exemplary embodiments enable any one of the users 101 to 104 to optionally switch the display mode of the virtual object 105, allowing the specific user 101 to explicitly share with other users 102 to 104 the area that the specific user 101 wants the other users 102 to 104 to focus on.

[0084] In the exemplary embodiments described above, while an example of a mixed reality space in which multiple users gather in the same real space to view virtual objects has been described, multiple users may be located in different real spaces, far apart from each other. In this case, it is conceivable that the exemplary embodiments are each applied to a system in which the user's HMD or a personal computer connected to the HMD is connected to each other via the Internet, and the user can exchange images and sounds captured by the HMD's camera and communicate with each other via a server. The respective HMDs of the users in different real spaces display overlaid images, in which the virtual objects are superimposed on the images captured in the corresponding real spaces. Therefore, multiple users far apart from each other can view the virtual objects via the network. At this time, based on the display mode set to presenter viewpoint sharing mode or optimal viewpoint sharing mode and instructions from a particular user regarding the virtual objects, other users can share the viewpoint position and orientation of the virtual objects.

[0085] In the above exemplary embodiments, although an example of each user wearing an HMD to view an overlay image of virtual objects superimposed on a real-world image has been described, the mobile terminal used by each user is not limited to an HMD, and may be a smartphone or tablet terminal.

[0086] Other embodiments

[0087] The embodiments of this disclosure can also be implemented by providing software (programs) that perform the functions of the above embodiments to a system or device via a network or various storage media, and the computer or central processing unit (CPU) or microprocessor unit (MPU) of the system or device reads and executes the program.

[0088] While this disclosure has been described with reference to exemplary embodiments, it should be understood that this disclosure is not limited to the disclosed exemplary embodiments. The scope of the appended claims is to be interpreted in the broadest possible sense in order to cover all such modifications and equivalent structures and functions.

Claims

1. An information processing device, comprising: The instruction acquisition unit is configured to acquire instructions from a first user wearing a first head-mounted display for virtual objects in a mixed reality space. A mode determination unit is configured to determine the display mode of a second head-mounted display worn by a second user based on the acquired instructions. A viewpoint determination unit is configured to determine a viewpoint for the virtual object in the mixed reality space based on a determined display mode. as well as An image generation unit is configured to generate an image comprising a mixed reality space of the virtual object as seen from a determined viewpoint. The mode determination unit, in response to the instruction, determines whether to use a first display mode or a second display mode. In the first display mode, the viewpoint of the first user relative to the virtual object is determined to be the viewpoint of the second user relative to the virtual object. In the second display mode, a pre-set viewpoint relative to the virtual object is determined to be the viewpoint of the second user relative to the virtual object. The image generation unit generates an image of the virtual object seen from a determined viewpoint different from that of the second user, superimposed on the image of the real space obtained by the second head-mounted display, as the mixed reality space image to be displayed on the second head-mounted display.

2. The information processing device according to claim 1, further comprising a position and posture acquisition unit, the position and posture acquisition unit being configured to acquire the position and posture of the virtual object in the mixed reality space, as well as the position and posture of the first user and the second user. in, The viewpoint determination unit determines the viewpoint for the virtual object corresponding to the display mode based on the position and posture of the virtual object, the respective positions and postures of the first user and the second user, and the instructions given to the virtual object. The image generation unit generates an image that overlays the image of the virtual object seen from the determined viewpoint onto the image of the real space obtained for each of the first user and the second user, as the image of the mixed real space.

3. The information processing device according to claim 2, in, The viewpoint determination unit determines the viewpoint for the virtual object for each of the first user and the second user, and The image generation unit generates an image that is superimposed on the image of the virtual object seen from the viewpoint determined for each of the first user and the second user, onto the image of the real space obtained for the corresponding user of the first user and the second user, as a mixed real space image for the corresponding user of the first user and the second user.

4. The information processing device according to claim 1, wherein, When the mode determination unit determines that the first display mode is to be used, the viewpoint determination unit determines the position and direction of the first user's viewpoint on the virtual object as the position and direction of the second user's viewpoint on the virtual object.

5. The information processing device according to claim 4, wherein, When the first user's gaze leaves the virtual object, the viewpoint determination unit determines the position and direction of the first user's viewpoint when looking at the virtual object as the position and direction of the viewpoint relative to the virtual object.

6. The information processing device according to claim 4, wherein, When the first user's gaze leaves the virtual object, the viewpoint determination unit determines the position and direction of the pre-set viewpoint as the position and direction of the viewpoint relative to the virtual object.

7. The information processing device according to claim 1, wherein, When the mode determination unit determines that the second display mode is to be used, the viewpoint determination unit determines the position and direction of the viewpoint that is preset for the virtual object as at least the position and direction of the second user's viewpoint for the virtual object.

8. The information processing device according to claim 1, wherein, The mode determination unit determines the display mode by switching between the first display mode and the second display mode based on instructions from the first user.

9. The information processing device according to claim 1, wherein, The mode determination unit determines whether to use the first display mode or the second display mode for each of the first user and the second user.

10. The information processing device according to claim 1, wherein, The viewpoint determination unit determines the position and direction of a predetermined arrangement of viewpoints for the virtual object as the viewpoint for the virtual object.

11. The information processing device according to claim 10, wherein, The viewpoint determination unit determines the positions and orientations of a plurality of viewpoints arranged in the predetermined manner for the virtual object as viewpoints for the virtual object, based on the position and orientation of the viewpoints used as references.

12. The information processing device according to claim 11, wherein, The position and orientation of the viewpoint used as the reference are the position and orientation of the viewpoint of the user used as the reference among the first user and the second user, or the position and orientation of a pre-set viewpoint.

13. The information processing device according to claim 1, wherein, The image generation unit generates an image of the mixed reality space with the virtual images of the first user and the second user superimposed on it, based on the position and direction of the viewpoints of the first user and the second user in the mixed reality space.

14. The information processing device according to claim 1, further comprising an operation unit configured to enable a user to perform independent operations on the virtual object.

15. The information processing device according to claim 1, wherein, The mixed reality space image generated by the image generation unit is displayed on the user's mobile terminal.

16. An information processing method, comprising: Obtain instructions for virtual objects in the mixed reality space from the first user wearing the first head-mounted display; Based on the acquired instructions, determine the display mode of the second head-mounted display worn by the second user; Based on the determined display mode, determine the viewpoint for the virtual object in the mixed reality space; as well as Generate an image of the mixed reality space that includes the virtual objects as seen from the determined viewpoint. In response to the instruction, it is determined whether to use a first display mode or a second display mode. In the first display mode, the viewpoint of the first user relative to the virtual object is determined to be the viewpoint of the second user relative to the virtual object. In the second display mode, a pre-set viewpoint relative to the virtual object is determined to be the viewpoint of the second user relative to the virtual object. Specifically, an image is generated that overlays an image of the virtual object seen from a determined viewpoint different from that of the second user onto an image of the real space obtained by the second head-mounted display, as the mixed reality space image to be displayed on the second head-mounted display.

17. A non-transitory computer-readable storage medium storing a program for causing a computer to perform an information processing method, the information processing method comprising: Obtain instructions for virtual objects in the mixed reality space from the first user wearing the first head-mounted display; Based on the acquired instructions, determine the display mode of the second head-mounted display worn by the second user; Based on the determined display mode, determine the viewpoint for the virtual object in the mixed reality space; as well as Generate an image of the mixed reality space that includes the virtual objects as seen from the determined viewpoint. In response to the instruction, it is determined whether to use a first display mode or a second display mode. In the first display mode, the viewpoint of the first user relative to the virtual object is determined to be the viewpoint of the second user relative to the virtual object. In the second display mode, a pre-set viewpoint relative to the virtual object is determined to be the viewpoint of the second user relative to the virtual object. Specifically, an image is generated that overlays an image of the virtual object seen from a determined viewpoint different from that of the second user onto an image of the real space obtained by the second head-mounted display, as the mixed reality space image to be displayed on the second head-mounted display.

18. A computer program product comprising a computer program that, when executed by a computer, implements the steps of the information processing method of claim 16.

Images