Mixed reality display system

Abstract

To enable a user to smoothly perform switching between a shared virtual object and a private virtual object without confusion.SOLUTION: In a mixed reality display system, a server 1 and a plurality of mixed reality display terminals (hereinafter referred to as terminals) 3a, 3b are connected over a network, and virtual objects are displayed so as to be mixed in the real space. The virtual objects include a shared virtual object for which a plurality of terminals have an operation authority and a private virtual object for which only a specific terminal has an operation authority. The server has virtual object attribute information for displaying the virtual objects on each terminal, and each terminal has a motion detecting unit that detects a motion of a user for switching between the shared virtual object and the private virtual object. When a result detected by the motion detecting unit is received from the terminal, the server updates the virtual object attribute information depending on whether the virtual object is the shared virtual object or the private virtual object, and transmits data of the virtual object after the update to each terminal.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present invention relates to a mixed reality display system and a mixed reality display terminal that generate and display a mixed reality space that combines virtually generated video, audio, etc. with real-world video, audio, etc., and allows users to operate virtually generated objects within the mixed reality space. [Background technology]

[0002] As background art in this technical field, Patent Document 1 discloses a system for displaying shared virtual objects and private virtual objects in a mixed reality environment. The shared virtual object allows multiple users to interact cooperatively, while the private virtual object is visible to a single user. The patent document states that providing a private virtual object makes it possible for multiple users to easily interact cooperatively with the shared virtual object. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Special Publication No. 2016-525741 Summary of the Invention [Problem to be solved by the invention]

[0004] The following are examples of specific uses of shared virtual objects (hereinafter referred to as shared virtual objects) and private virtual objects (hereinafter referred to as personal virtual objects).

[0005] The first example is when a prototype model or the like is displayed as a shared virtual object in a mixed reality space and the design of the model is discussed. Sharing a virtual object among multiple designers allows for smooth discussion among the multiple designers. However, since the authority to modify the design of a shared virtual object is given to all designers, it is not suitable for cases where individual designers want to individually consider modifications. In such cases, it would be convenient to be able to switch to a personal virtual object that a specific designer is given the authority to modify. To do this smoothly, it is desirable to be able to copy a displayed shared virtual object as a personal virtual object for a specific user, and conversely, to be able to merge a personal virtual object for a specific user with a shared virtual object.

[0006] A second example is when advertisements are displayed as shared virtual objects, such as in a virtual flyer drop-off area. In this case, multiple users in the mixed reality space are provided with the experience of seeing advertisements displayed in the same location. In this case, as can be inferred from user actions in a real flyer drop-off area, a request arises from a user to copy the advertisements as their own personal copy. In such a case, it is desirable to be able to switch the shared virtual object into a personal virtual object for that user. That is, an operation for switching between a shared virtual object and a personal virtual object (here meaning a copy operation or a merge operation) is required.

[0007] However, the invention described in Patent Document 1 does not take into consideration the alternation between a shared state (shared virtual object) and a personal state (personal virtual object) for a certain virtual object as described above. Furthermore, even if switching between a shared virtual object and a personal virtual object is performed, the following problems must be overcome in order to perform this smoothly.

[0008] (1) Each user must remember whether a virtual object is a shared virtual object or a personal virtual object and use it accordingly, and users who do not remember this will become confused. (2) When a personal virtual object is displayed only to a specific user A and is not visible to other users B (for example, in the case of Patent Document 1), operations performed on the invisible personal virtual object are not recognized by other users B, and user B becomes confused because he does not know what operations user A is performing. (3) A simple method is required for users to easily switch between shared virtual objects and personal virtual objects (by copy or merge operations). (4) When user A is using a personal virtual object to operate a shared virtual object, the operation may be hidden by user A's hand and may not be visible to other user B.

[0009] An object of the present invention is to provide a mixed reality display system and a mixed reality display terminal that allow a user to smoothly switch between shared virtual objects and personal virtual objects in a mixed reality space without becoming confused. [Means for solving the problem]

[0010] The present invention provides a mixed reality display system in which a server and multiple mixed reality display terminals used by multiple users are connected via a network and display virtual objects in a composite form in real space, wherein the virtual objects include shared virtual objects for which operation authority is granted to the multiple mixed reality display terminals and personal virtual objects for which operation authority is granted only to specific mixed reality display terminals. The server has virtual object attribute information for displaying the virtual objects on the multiple mixed reality display terminals, and each of the multiple mixed reality display terminals has a motion detection unit that detects a user's motion for switching between the shared virtual object and the personal virtual object. Upon receiving a detection result from the motion detection unit from the mixed reality display terminal, the server updates the virtual object attribute information depending on whether the virtual object is the shared virtual object or the personal virtual object, and transmits the updated virtual object data to the multiple mixed reality display terminals. [Effects of the Invention]

[0011] According to the present invention, the user can smoothly switch between a shared virtual object and a personal virtual object in a mixed reality space without becoming confused. [Brief explanation of the drawings]

[0012] [Figure 1A] 1 is a block diagram showing the overall configuration of a mixed reality display system according to a first embodiment. [Figure 1B] FIG. 1 is a diagram showing an example of a mixed reality space in which shared virtual objects and personal virtual objects are mixed. [Figure 2A] FIG. 2 is a diagram showing the hardware configuration of mixed reality display terminals 3a and 3b. [Figure 2B] FIG. 2 is a diagram showing the software configuration of the mixed reality display terminals 3a and 3b. [Figure 3A] FIG. 2 shows the hardware configuration of the server 1. [Figure 3B] FIG. 2 is a diagram showing the software configuration of the server 1. [Figure 4A] 10 is a data table showing an example of virtual object attribute information 239. [Figure 4B] 10 is a data table showing an example of user authentication information 240. [Figure 4C] 10 is a data table showing an example of device management information 241. [Figure 5] FIG. 3 is a diagram showing an operation sequence in the first embodiment. [Figure 6A] A diagram showing the personalization of a virtual object (view from the user's perspective during operation). [Figure 6B] A diagram showing the personalization of a virtual object (as seen by other users). [Figure 7A] FIG. 10 is a diagram showing the display state of a virtual object during a personalization operation (viewed from the user during operation, display pattern P1). [Figure 7B] FIG. 10 is a diagram showing the display state of a virtual object during a personalization operation (viewed from the user during operation, display pattern P2). [Figure 7C] FIG. 10 is a diagram showing the display state of a virtual object during a personalization operation (as seen from another user, display pattern P3). [Figure 7D]FIG. 10 is a diagram showing the display state of a virtual object during a personalization operation (as seen from another user, display pattern P4). [Figure 8] FIG. 10 is a diagram illustrating differences in the display positions of personal virtual objects (display patterns P3 and P4). [Figure 9A] FIG. 10 is a diagram showing the sharing operation of a virtual object (as seen from the user during operation). [Figure 9B] FIG. 10 is a diagram showing the sharing of a virtual object (as seen from another user's perspective). [Figure 10A] FIG. 10 is a diagram showing a display state of a virtual object in a sharing operation (a view seen from the user during operation, display pattern P1). [Figure 10B] FIG. 10 is a diagram showing the display state of a virtual object in a sharing operation (as seen from another user, display pattern P3). [Figure 10C] FIG. 10 is a diagram showing a display state of a virtual object in a sharing operation (viewed from another user, display pattern P4). [Figure 11] 10A and 10B are diagrams illustrating the shape determination of a virtual object in a sharing operation. [Figure 12] FIG. 10 is a block diagram showing the overall configuration of a mixed reality display system according to a second embodiment. [Figure 13A] FIG. 10 is a diagram showing the hardware configuration of a mixed reality display terminal (with built-in server function) 3c. [Figure 13B] FIG. 10 is a diagram showing the software configuration of a mixed reality display terminal (with built-in server function) 3c. [Figure 14] FIG. 10 is a diagram showing an operation sequence in the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0013] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. [Example]

[0014] FIG. 1A is a block diagram showing the overall configuration of a mixed reality display system according to a first embodiment. The mixed reality display system 10 of this embodiment is configured by connecting a server 1 and multiple mixed reality display terminals 3a and 3b for users to a network 2. Here, two mixed reality display terminals are used, but it goes without saying that the system may be configured with two or more terminals. An example of a mixed reality display terminal is a head-mounted display (HMD) that is worn on the user's head. Hereinafter, the mixed reality display terminal will be simply referred to as a "terminal."

[0015] Server 1 has the functions of retaining attribute information of virtual objects, performing device authentication, and performing user authentication. It also has a communication I / F and the function of communicating with mixed reality display terminals (terminals) 3a and 3b via network 2. Server 1 has a video and audio output function, which outputs the attribute information of virtual objects retained by the server to terminals 3a and 3b, and also has an input function, which allows it to modify programs and functions within the server.

[0016] On the other hand, the mixed reality display terminals (terminals) 3a and 3b are equipped with a communication I / F, receive user authentication from the server 1 via the network 2, and further receive the virtual object data and user authentication information held in the server 1. The terminals 3a and 3b are equipped with a video output unit and an audio output unit, and have the function of outputting and displaying the video and audio of the virtual object based on the virtual object data received from the server 1, and further modifying them.

[0017] In this case, the terminals 3a and 3b have a function of appropriately adjusting and outputting the video and audio so that when the user perceives the output video and audio, the user can perceive it as if a virtual object or virtual audio were present at a specific position in real space. For this purpose, the terminals 3a and 3b have means for detecting their position and acceleration information, and have a function of dynamically generating video and audio using the position and acceleration information.

[0018] FIG. 1B is a diagram showing an example of a mixed reality space in which shared virtual objects and personal virtual objects are mixed, provided by the mixed reality display system 10 of FIG. 1. Two users 4a and 4b are wearing head-mounted devices (HMDs) 3a and 3b, respectively, and are manipulating virtual objects in the mixed reality space. A shared virtual object 7 and a personal virtual object 8 are displayed in the field of view 5a of user 4a via terminal 3a. Similarly, a shared virtual object 7 and a personal virtual object 8 are displayed in the field of view 5b of user 4b via terminal 3b. User 4a is performing some kind of operation on the personal virtual object 8.

[0019] Next, the internal configurations of the mixed reality display terminals 3a and 3b and the server 1 will be described. [Internal structure of a mixed reality display device] 2A is a diagram showing the hardware configuration of the mixed reality display terminals 3a and 3b. Here, the terminal 3a and the terminal 3b are described as having the same configuration, but they do not necessarily have to have the same configuration.

[0020] The distance sensor 200 detects the user's hand movements and the distance to objects in real space. The gyro sensor 201 and acceleration sensor 202 accurately measure the position and acceleration of the terminal. The communication I / F 203 communicates with the server 1 via the network 2. Furthermore, the in-camera 204 detects the user's line of sight, and the out-camera 205 captures images of real space.

[0021] The control unit 209 reads various data 212 and programs 213 stored in the storage 211 through a various program function unit 214 provided in the memory 210, thereby integrating the aforementioned various sensor and camera images to generate video and audio of a virtual object. Then, the control unit 209 adjusts the video and audio of the virtual object so that it exists at a specific position and direction in real space.

[0022] The video and audio of the generated virtual object are output and displayed from a video output unit 206 and an audio output unit 207. An input unit 208 accepts tactile or audio input from the user. The above blocks are connected by a bus 215.

[0023] This allows a user who sees the video or hears the audio on the terminals 3a and 3b to perceive a virtual object or virtual audio as if it were located at a specific position in real space. Note that many technologies have been published so far regarding technology for generating the video and audio of a virtual object so that it appears to be located at a specific position and direction in real space (mixed reality, MR), and one example is disclosed in Patent Document 1.

[0024] 2B is a diagram showing the software configuration of the mixed reality display terminals 3a and 3b, illustrating the configuration of the memory 210 and storage 211 in FIG. 2A.

[0025] The storage 211 stores various data 302, a gaze direction detection program 303, a face position / direction detection program 304, a video generation program 305, a voice generation program 306, and a communication control program 307. These data and programs are called into memory by a device control unit 301 provided in the memory 210, and various functions are realized. Note that these data and programs may be stored in the memory 210 in advance.

[0026] In the following description, for simplicity, it is assumed that the various functions realized by the device control unit 301 executing the programs are mainly realized by the various program function units.

[0027] The gaze direction detection program 303 detects the gaze direction of the user based on image information obtained from the in-camera 204. Hereinafter, the gaze direction detection program 303 and the in-camera 204 will also be referred to as a "gaze detection unit."

[0028] The motion detection program 308 detects motions performed by the user in real space based on information obtained from the distance sensor 200, the outer camera 205, etc. Motions here include, for example, moving a hand from a specific position to a specific position, closing a hand, opening a hand, etc. Many methods for detecting a user's motions from distance sensor information or camera information have been known in the past, and one example is disclosed in Reference 1.

[0029] [Reference 1] Zhou Ren, et al. “Robust hand gesture recognition with kinect sensor.” Proceedings of the 19th ACM international conference on Multimedia. ACM, 2011. The motion detection program 308 may further include a function to detect "voice motion" from the volume and content of speech of the voice input to the input means 208. Note that speech recognition technology can be used as a method for detecting the content of speech from the voice. Many voice recognition techniques have been known in the past, and one example is disclosed in Reference 2.

[0030] [Reference 2] Lawrence R. Rabiner and Biing-Hwang Juang. “Fundamentals of speech recognition.” (1993). Hereinafter, the distance sensor 200, the outer camera 205, the input means 208, and the motion detection program 308 will also be referred to as a "motion detection unit."

[0031] A face position / direction detection program 304 detects the position and direction of the user's face in the space based on signals from the gyro sensor 201 and acceleration sensor 202. This information is used by a video generation program 305 and a sound generation program 306 when generating virtual objects and virtual sounds.

[0032] The video generation program 305 generates a video to be output from the video output unit 206 in accordance with the information on the position and direction of the face obtained by the face position / direction detection program 304 and the image obtained from the in-camera 204. This video is adjusted so that the user perceiving it perceives it as if a virtual object were present at a specific position in real space.

[0033] The audio generation program 306 generates audio to be output from the audio output unit 207 in accordance with the information on the position and direction of the face obtained by the face position / direction detection program 304 and the image obtained from the in-camera 204. This audio is adjusted so that the user perceiving it perceives it as if a virtual object were present at a specific position in real space.

[0034] The communication control program 307 communicates with the server 1 via the communication I / F 203. The contents communicated here include attribute information (data) of virtual objects and user authentication information, but other information may also be included.

[0035] [Internal configuration of the server] FIG. 3A is a diagram showing the hardware configuration of the server 1. As shown in FIG. The storage 235 stores various data 237 and programs 238 for operating the server, as well as virtual object attribute information 239 for controlling the terminals 3 a and 3 b, user authentication information 240, and device management information 241. The data and programs are read by the control unit 233 into the memory through a various program function unit 236 provided in the memory 234 and executed.

[0036] The communication I / F 231 communicates with the terminals 3a and 3b through the network 2. The contents communicated here include the data of the virtual object attribute information 239 and the user authentication information 240, but other data may also be included.

[0037] The input means 232 accepts user operation input and can also modify the internal data of the server 1 based on the user input. The video output unit 242 and the audio output unit 243 display the internal data of the server 1 (such as the virtual object attribute information 239) to the user. The above blocks are connected by a bus 244.

[0038] [Server software configuration] Fig. 3B is a diagram showing the software configuration of the server 1. Here, the configuration of the memory 234 and storage 235 in Fig. 3A is shown. The storage 235 stores various data 335, a user position / direction detection program 336, a communication control program 337, virtual object attribute information 239, user authentication information 240, and device management information 241. The memory 234 also stores a device control unit 331, a virtual object management unit 332, a user management unit 333, and a device management unit 334.

[0039] The virtual object management unit 332 manages the position, direction, shape, user operation authority, and the like of virtual objects in the mixed reality space based on the virtual object attribute information 239 stored in the storage 235. Then, the virtual object attribute information 239 is corrected based on information from each of the terminals 3 a and 3 b obtained by the communication I / F 231. It is also possible to correct the virtual object attribute information 239 based on user operation via the input means 232. Furthermore, the information described in the virtual object attribute information 239 can be displayed to the user from the video output unit 242 and the audio output unit 243.

[0040] The user management unit 333 manages user authentication information 240 for authenticating each user and device management information 241 for authenticating the terminal owned by the user. These pieces of information 240, 241 are stored in the storage 235. The user management unit 333 acquires user authentication information such as a user name and password from user input at each terminal 3a, 3b (input means 208) via the communication I / F 231, and checks whether the user is a legitimate registered user. If the user authentication information is incorrect, it returns error information. Note that the user name and password are stored in advance in the storage 211 of each terminal 3a, 3b, so that authentication can be performed without the user having to input them each time.

[0041] The device management unit 334 sends information required for communication with the terminals 3 a and 3 b to the communication control program 337 in accordance with the device management information 241 stored in the storage 235 .

[0042] Each piece of data and programs stored in the storage 235 is read into the memory 234 and executed under the control of the device control unit 331 stored in the memory 234. Note that each piece of data and programs stored in the storage 235 may be stored in the memory 234 in advance.

[0043] In the following explanation, for simplicity, it will be assumed that the various functions realized by the device control unit 331 executing the programs are mainly realized by the various program function units.

[0044] The user position / direction detection program 336 detects the position and direction of each of the terminals 3a and 3b based on information obtained from each of the terminals 3a and 3b via the communication I / F 231. This information may be transmitted to each of the terminals 3a and 3b.

[0045] The communication control program 337 controls communication with the terminals 3a and 3b. The contents communicated here are the virtual object attribute information 239 and the user authentication information 240, but other information may also be included.

[0046] [Data format] The data formats of various types of information stored in the server 1 will be described below. 4A is a data table showing an example of virtual object attribute information 239 managed by the virtual object management unit 332 of the server 1. The virtual object attribute information 239 describes information such as a virtual object ID 401, a virtual object name 402, a virtual object direction 403, a virtual object position 404, a virtual object color and shape 405, a copy source 406 and a copy destination 407 that indicate a copy relationship between virtual objects, and a user operation authority 408. Note that a shared / personal item 409 is provided for illustrative purposes.

[0047] The virtual object name 402, virtual object direction 403, virtual object position 404, and virtual object color and shape 405 are data for displaying the virtual objects on terminals 3a and 3b. In this example, virtual object IDs 2 and 3 are "shared" virtual objects for which operation authority 408 is given to all users or a specific group. Virtual object IDs 1 and 4 are "personal" virtual objects for which operation authority 408 is given to specific user 1. Personal virtual object ID 4 indicates that it has been copied from shared virtual object ID 3.

[0048] 4B is a data table showing an example of user authentication information 240 managed by the user management unit 333 of the server 1. The user authentication information 240 includes a user ID (management number) 411, a user password 412, and the like. In addition to these, the user name, name, address, contact information, and the like may also be included. In addition to user authentication using the password 412, general biometric authentication such as finger vein, fingerprint, voice, face image, or iris may also be used, and in such cases, the items in the user authentication information 240 are changed as appropriate.

[0049] 4C is a data table showing an example of device management information 241 managed by the device management unit 334 of the server 1. The device management information 241 describes a device ID (management number) 421, a device name 422, a device password 423, a device IP address 424, etc. of each terminal.

[0050] [Operation sequence] FIG. 5 is a diagram showing an operation sequence in Example 1. Here, communication between the server 1 and the terminal 3a (user 4a) and terminal 3b (user 4b) when a "personalization operation" and a "sharing operation" of a virtual object are described. The "personalization operation" here refers to copying a shared virtual object to generate a personal virtual object, thereby switching it into a virtual object that only a specific user has operation authority over. On the other hand, the "sharing operation" refers to merging a personal virtual object with a shared virtual object, thereby switching it into a new shared virtual object that each user has operation authority over.

[0051] When the server 1, mixed reality display terminal 3a, and mixed reality display terminal 3b are started up, virtual object data is generated in the server 1 according to the virtual object attribute information 239 (sequence S101). When user authentication information (or device management information) is sent from terminal 3a (user 4a) to the server 1, it is checked against user authentication information 240 (or device management information 241) in the server 1. If authentication is successful, the virtual object data is sent to terminal 3a. In terminal 3a, based on the received virtual object data, an image and sound of the virtual object are generated by the image generation program 305 and the sound generation program 306, and are output and displayed from the image output unit 206 and the sound output unit 207 (sequence S102).

[0052] On the other hand, user authentication is also performed on terminal 3b (user 4b), and if the authentication is successful, the image and sound of the virtual object are output and displayed on terminal 3b (sequence S103). The virtual object displayed here is a shared virtual object that both user 4a and user 4b have the operation authority to.

[0053] Next, when the user 4a performs an action on the virtual object displayed, for example, by pulling it toward his / her hand, the motion detection unit of the terminal 3a (a sensor such as the outer camera 205 and the motion detection program 308) detects this motion as a personalization motion (sequence S104). The terminal 3a then notifies the server 1 that it has detected the personalization motion. The server 1 updates the virtual object attribute information 239 in accordance with the personalization motion, copies the shared virtual object, and generates data for a new personal virtual object (sequence S105). The personal virtual object generated here is one that only the user 4a (terminal 3a) has the authority to operate.

[0054] The data of the new personal virtual object is transmitted to terminal 3a and terminal 3b, and terminals 3a and 3b modify the display state of the currently displayed virtual object based on the received virtual object data (sequences S106 and S107). As will be described later, the display states of the virtual objects differ between terminal 3a and terminal 3b, and these are managed by the virtual object attribute information 239 of server 1.

[0055] Furthermore, when user 4a performs an action such as moving the personal virtual object closer to the shared virtual object while the virtual object is displayed, the action detection unit of terminal 3a detects this action as a sharing action (sequence S108). Terminal 3a then notifies the server 1 that it has detected the sharing action. The server 1 updates the virtual object attribute information 239 in accordance with the sharing action, and merges the personal virtual object and the shared virtual object to generate data for a new shared virtual object (sequence S109). The shared virtual object generated here is one that both user 4a and user 4b have permission to operate.

[0056] The data of the new shared virtual object is transmitted to terminal 3a and terminal 3b, and each of terminals 3a and 3b modifies the display state of the currently displayed virtual object based on the received virtual object data (sequences S110 and S111).

[0057] [Personalization of Shared Virtual Objects] The behavior when the personalization operation (FIG. 5, S104) is performed will be described below with reference to FIGS.

[0058] 6A and 6B are diagrams showing a personalization operation in which a shared virtual object 501 is switched to a personal virtual object 502 in a mixed reality space. Here, an example is shown in which user 4a performs a personalization operation on the shared virtual object 501 to copy a personal virtual object 502 from the shared virtual object 501. The personalization operation in this embodiment means leaving the shared virtual object 501 as is, and generating a personal virtual object 502 that only user 4a has the right to operate.

[0059] The motion detection units (distance sensor 200, outer camera 205, input means 208) of the mixed reality display terminals 3a and 3b determine that a particular motion performed by the user, for example, as follows, is a personalized motion. - Make a specific shape with your fingers to attract a shared virtual object. For example, you can make a gesture with your index finger and thumb touching to attract a shared virtual object. - Make a specific hand gesture and then pull the shared virtual object towards you. For example, move your hand around the periphery of the shared virtual object and then pull it towards you. - Say "copy" and then pull the shared virtual object towards you. - Attract a shared virtual object and then say "copy." While pulling a shared virtual object, say "copy."

[0060] A mechanism may be provided that allows a user to register a specific action when performing a personalization action. Whether or not the shared virtual object 501 is copied to a personal virtual object 502 by the above action (whether or not copying is permitted) may be described in the virtual object attribute information 239. Whether or not the virtual object is personalized may then be controlled according to the copy permission information.

[0061] FIG. 6A shows the mixed reality space 5a as seen by user 4a through terminal 3a during the personalization operation. Meanwhile, FIG. 6B shows the mixed reality space 5b as seen by another user 4b through terminal 3b. As shown in FIGS. 6A and 6B, a shared virtual object 501 and a personal virtual object 502 copied from it can be seen by both users 4a and 4b, but are displayed in different ways (e.g., color, shape, distance, sound, etc.) for users 4a and 4b. This allows users 4a and 4b to easily distinguish between the shared virtual object 501 and the personal virtual object 502. Specific display states when transitioning from the shared virtual object 501 to the personal virtual object 502 are described below with reference to FIGS. 7A to 7D.

[0062] 7A and 7B show the display state of the virtual object as seen by the operating user 4a when transitioning from a shared virtual object 501 to a personal virtual object 502. Here, it is assumed that virtual sound 503 is being output from the shared virtual object 501. It is also assumed that the display state transitions in the order of timings (1), (2), and (3) in the figures. The arrows in the figures indicate the movement of the user's hand (white arrow) and the movement of the display position of the virtual object (gray arrow).

[0063] FIG. 7A shows the case according to <Display Pattern P1>. In this case, the display position of the shared virtual object 501 is fixed at its original position during the transition to personalization. At timing (1), the user 4a places his / her hand on the shared virtual object 501. At timing (2), when the user 4a performs an action of pulling the shared virtual object 501 toward his / her hand, the motion detection unit of the terminal 3a determines this as a personalization action and notifies the server 1. The server 1 copies the shared virtual object 501 to generate a new personal virtual object 502 and sends it to the terminal 3a. The terminal 3a displays the personal virtual object 502, but its display position is adjusted to match the position of the hand detected by the motion detection unit. Finally, at timing (3), the personal virtual object 502 is displayed as if it is being pulled toward the hand of the user 4a.

[0064] From timing (1) to timing (3), the color of the shared virtual object 501 gradually becomes lighter and the volume of its virtual sound 503 gradually decreases. On the other hand, the color of the newly generated personal virtual object 502 gradually becomes darker and the volume of the virtual sound 503 is also adjusted to increase. This allows the user 4a to easily switch from the shared virtual object 501 to the personal virtual object 502 and easily distinguish between the shared virtual object 501 and the personal virtual object 502 generated from it.

[0065] 7B shows the case according to <display pattern P2>. In this case, the display position of the shared virtual object 501 is moved according to the distance between the position of the hand of the user 4a and the shared virtual object 501. The differences from <display pattern P1> in FIG. 7A will be described.

[0066] At timing (2), when the user 4a performs a personalization action, the terminal 3a displays the personal virtual object 502 generated by the server 1 in accordance with the position of the hand of the user 4a. At this time, the shared virtual object 501 is displayed by moving it to the middle between the original position of the shared virtual object 501 and the position of the hand (i.e., it is moved from the original position closer to the position of the hand).

[0067] At timing (3), when the position of the hand moves away from the original position of the shared virtual object 501 by a certain distance or more, the shared virtual object 501 is displayed returned to its original position. The personal virtual object 502 is displayed as if it is being drawn to the hand of the user 4a, as in FIG. 7A.

[0068] 7C and 7D show the display state as seen by a user 4b other than the operating user 4a when transitioning from a shared virtual object 501 to a personal virtual object 502. Timings (1), (2), and (3) in the figures correspond to timings (1), (2), and (3) in Figures 7A and 7B, respectively.

[0069] 7C shows the case according to <display pattern P3>. In this case, the position of the generated personal virtual object 502 visible to the other user 4b is set to the same position as that visible to user 4a. In the mixed reality space 5b visible to user 4b, the personal virtual object 502 generated by user 4a at timings (2) and (3) is displayed semi-transparently, and no virtual sound is output. If the distance from the shared virtual object 501 to the personal virtual object 502 at timing (3) is d3, this is equal to the position visible to user 4a (distance d1 in FIG. 7A or 7B).

[0070] On the other hand, the shared virtual object 501 is displayed normally at timings (1) and (2), but at timing (3) after the personal virtual object 502 is generated by user 4a, it is displayed differently from the normal display, for example in a color darker than the normal color. This allows the other user 4b to easily recognize that the personal virtual object 502 has been copied from the shared virtual object 501, and does not cause confusion.

[0071] Note that changing the color of the shared virtual object 501 at timing (3) is merely an example and is not limiting. Various other display states are possible, such as displaying the outline in red, blinking, displaying a mark indicating a copy, or changing the brightness.

[0072] 7D shows the case according to <display pattern P4>. In this case, the position of the generated personal virtual object 502 that is visible to other user 4b is made different from the position that is visible to user 4a. That is, at timing (3), the personal virtual object 502 generated by user 4a is displayed at a position away from the position of user 4a's hand. The distance from the shared virtual object 501 to the personal virtual object 502 at this time is set to d4. This causes user 4b to perceive the personal virtual object 502 as being in mid-air.

[0073] Figure 8 is a diagram illustrating the difference in the display position of the personal virtual object 502 between Figure 7C and Figure 7D. The horizontal axis represents the display position of the personal virtual object 502 as seen by the operating user 4a (distance d1 from the shared virtual object 501), and the vertical axis represents the display position of the personal virtual object 502 as seen by the other user 4b (distances d3 and d4 from the shared virtual object 501).

[0074] In <display pattern P3> of Fig. 7C, the display position (d3) of the personal virtual object 502 as seen by the other user 4b is equal to the display position (d1) of the personal virtual object 502 as seen by the operating user 4a. In contrast, in <display pattern P4> of Fig. 7D, the display position (d3) of the personal virtual object 502 as seen by the other user 4b is made smaller than the display position (d1) of the personal virtual object 502 as seen by the operating user 4a.

[0075] According to the display pattern P4 in FIG. 7D, at timing (3) when the user 4a generates the personal virtual object 502, the personal virtual object 502, which is in the user's hand to the user 4a, is perceived by the user 4b as being in mid-air.

[0076] The advantage of the display method of <Display Pattern P4> is that while user 4a is manipulating the personal virtual object 502 with his / her hand, user 4b can observe the personal virtual object 502 without it being hidden by user 4a's hand. In addition, it is easier to identify that the personal virtual object 502 has been generated by copying it from the shared virtual object 501, and confusion does not occur.

[0077] In FIG. 7C or FIG. 7D, the following method can be used to reliably notify the other user 4b that the shared virtual object 501 has been switched to the personal virtual object 502.

[0078] At the time when user 4a performs a personalization action, user 4b may be looking in a different direction. In this case, it is preferable to detect the gaze direction of user 4b using a gaze detection unit (in-camera 204, gaze detection program 303) provided in terminal 3b of user 4b, and switch the display state (e.g., color) of shared virtual object 501 at the timing when user 4b's gaze is placed on the shared virtual object 501. This prevents user 4b from missing the personalization switch of shared virtual object 501.

[0079] Furthermore, if the position where user 4a performed the personalization operation is far from the position of user 4b, it is preferable to detect the distance between the shared virtual object 501 and terminal 3b using distance sensor 200, and switch the display state (e.g., color) of the shared virtual object 501 when the distance approaches a predetermined distance. This prevents user 4b from missing the personalization switch of the shared virtual object 501.

[0080] It is also preferable to use the acceleration sensor 202 to detect the speed or acceleration at which the user 4b (terminal 3b) is approaching the shared virtual object 501, and to switch the display state (e.g., color) when the speed or acceleration exceeds a predetermined value. This prevents the user 4b from missing the fact that the shared virtual object 501 has been switched to a personal virtual object when the user approaches the shared virtual object 501.

[0081] [Sharing personal virtual objects] The behavior when a sharing action is detected (FIG. 5, S108) will be described below with reference to FIGS.

[0082] 9A and 9B are diagrams showing a sharing operation in which a personal virtual object 601 is switched to a shared virtual object 602 in a mixed reality space. Here, an example is shown in which user 4a performs a sharing operation on the personal virtual object 601, thereby merging the personal virtual object 601 into the shared virtual object 602. The sharing operation in this embodiment refers to generating a new shared virtual object 602 from the personal virtual object 601 and the shared virtual object 602, thereby reflecting the characteristics of the personal virtual object 601 and giving operation authority to another user 4b through sharing.

[0083] The motion detection units of the mixed reality display terminals 3a and 3b determine that a specific motion performed by the user, for example, as follows, is a shared motion. - Make a specific shape with your fingers and move the personal virtual object closer to the shared virtual object. For example, bring the personal virtual object closer while holding your index finger and thumb together. - Make a specific hand gesture and then move the personal virtual object closer to the shared virtual object. For example, move your hand around the periphery of the personal virtual object and then move it closer to the shared virtual object. Say "merge" and then move your personal virtual object closer to the shared virtual object. - Move your personal virtual object close to the shared virtual object, then say "merge." While moving a personal virtual object closer to a shared virtual object, say "merge." A mechanism may be provided that allows the user to register specific actions when performing shared actions.

[0084] FIG. 9A shows the mixed reality space 5a as seen by user 4a through terminal 3a during the sharing operation. Meanwhile, FIG. 9B shows the mixed reality space 5b as seen by another user 4b through terminal 3b. FIGS. 9A and 9B illustrate the operation of merging two virtual objects by moving a personal virtual object 601 closer to a shared virtual object 602. Again, the personal virtual object 601 and the shared virtual object 602 are displayed in different ways (e.g., color, shape, distance, sound, etc.) for users 4a and 4b. This allows users 4a and 4b to easily distinguish between the personal virtual object 601 and the shared virtual object 602. Specific display states when transitioning from the personal virtual object 601 to the shared virtual object 602 are described below with reference to FIGS. 10A to 10C.

[0085] 10A shows the display state of the virtual object as seen by the operating user 4a when transitioning from a personal virtual object 601 to a shared virtual object 602. Here, it is assumed that a virtual sound 603 is being output from the personal virtual object 601. It is also assumed that the display state transitions in the order of timings (1), (2), and (3) in the figure. The arrows in the figure indicate the movement of the user's hand (white arrow) and the movement of the display position of the virtual object (gray arrow). This display pattern corresponds to <Display Pattern 1> in FIG. 7A.

[0086] At timing (1), user 4a places his / her hand on personal virtual object 601. At timing (2), when user 4a performs an action to bring personal virtual object 601 closer to shared virtual object 602, the action detection unit of terminal 3a determines this to be a sharing action and notifies server 1. Server 1 merges personal virtual object 601 and its copy source shared virtual object 602 to generate a new shared virtual object 602 and sends it to terminal 3a. At timing (3), terminal 3a displays the new shared virtual object 602 in its original display state (dark color).

[0087] From timing (1) to (3), the color of the shared virtual object 602 gradually becomes darker and the volume of its virtual sound 604 increases. On the other hand, the color of the personal virtual object 601 gradually becomes lighter and the volume of the virtual sound 603 is also adjusted to decrease. This allows the user 4a to easily switch from the personal virtual object 601 to the shared virtual object 602.

[0088] 10B and 10C show the display state of the virtual object as seen by another user 4b when transitioning from a personal virtual object 601 to a shared virtual object 602. Timings (1), (2), and (3) in the figures correspond to timings (1), (2), and (3) in FIG. 10A, respectively.

[0089] 10B shows the case according to <display pattern P3>. In this case, as explained in FIG. 8, the position of the personal virtual object 601 that is visible to the other user 4b is set to be the same as the position that is visible to the user 4a.

[0090] At timing (1), in the mixed reality space 5b visible to user 4b, user 4a's personal virtual object 601 is displayed semi-transparently, and the shared virtual object 602 is displayed in a dark color. That is, the distance d3 from the shared virtual object 602 to the personal virtual object 601 is equal to the position visible to user 4a (distance d1 in FIG. 10A). At timing (2), user 4a moves the personal virtual object 601 closer to the shared virtual object 602, and at timing (3), the two virtual objects are merged, and the new shared virtual object 602 is displayed in a normal color.

[0091] This allows the user 4b to easily distinguish between the personal virtual object 601 and the shared virtual object 602, and to easily recognize that the shared virtual object 602 is a merged object of the personal virtual object 601, eliminating confusion.

[0092] 10C shows the case according to <display pattern P4>. In this case, as explained in FIG. 8, the position of the personal virtual object 601 visible to the other user 4b is made different from the position visible to the user 4a. That is, at timing (1), the distance d4 from the shared virtual object 602 to the personal virtual object 601 is made smaller than the position visible to the user 4a (distance d1 in FIG. 10A). This causes the user 4b to perceive the personal virtual object 601 as being in mid-air. Therefore, the user 4b can observe the personal virtual object 601 without it being hidden by the hand of the user 4a.

[0093] 11 is a diagram illustrating how the shape of a virtual object is determined in a sharing operation. Generally, a personal virtual object 601 and the shared virtual object 602 that merges it have different shapes. Therefore, when these virtual objects are merged, the user 4a can sequentially check whether or not to incorporate (combine) the parts of the shapes of the virtual objects that do not encompass each other into a new shared virtual object.

[0094] At timing (1), a personal virtual object 601 is assumed to have been copied from a shared virtual object 602. After copying, a new personal virtual object 601a and a personal virtual object 601b are generated by an operation of user 4a, and are combined with the personal virtual object 601. Here, an operation is performed to merge the collection of personal virtual objects 601, 601a, and 601b into the shared virtual object 602 from which they were copied.

[0095] At timing (2), the user 4a selects whether to combine the personal virtual object 601a, which is the first difference, with the shared virtual object 602. For this selection process, methods such as hand gestures and voice recognition are used. For example, it is assumed that the personal virtual object 601a is selected to be combined with the shared virtual object 602. Next, at timing (3), the user 4a selects whether to combine the personal virtual object 601b, which is the second difference, with the shared virtual object 602. It is assumed that the user 4a rejects the combination of the personal virtual object 601b. As a result, a new collection of shared virtual objects is generated in which the personal virtual object 601a is combined with the shared virtual object 602, as shown at timing (4).

[0096] Although Figure 11 describes the case of combining newly generated personal virtual objects, even if changes are made to the personal virtual object itself, the same procedure can be used to select whether or not to merge the personal virtual object with a shared virtual object.

[0097] As described above, according to the first embodiment, in the mixed reality space, switching between a shared virtual object and a personal virtual object can be performed smoothly without causing confusion for not only the operating user but also other users. [Example]

[0098] In the second embodiment, all of the functions performed by the server 1 in the first embodiment are configured to be built into the mixed reality display terminal. Therefore, the server 1 in the first embodiment is not required. Below, differences from the first embodiment will be described.

[0099] 12 is a block diagram showing the overall configuration of a mixed reality display system according to Example 2. The mixed reality display system 10′ of this example is composed of a network 2 and multiple mixed reality display terminals 3c and 3b for users. Among these, the mixed reality display terminal 3c is a terminal with a built-in server function.

[0100] 13A is a diagram showing the hardware configuration of a mixed reality display terminal (with built-in server function) 3c. The configuration of the terminal 3b is the same as that of FIG. 2A in the first embodiment. In the terminal 3c, virtual object attribute information 239, user authentication information 240, and device management information 241 that were provided in the server 1 are additionally stored in the storage 211. The role of each block is as described in FIG. 3A in the first embodiment.

[0101] Fig. 13B is a diagram showing the software configuration of the mixed reality display terminal (with built-in server function) 3c. The configuration of the terminal 3b is the same as that of Fig. 2B in Example 1. Here, the configuration of the memory 210 and storage 211 in Fig. 13A is shown.

[0102] In the terminal 3c, the virtual object management unit 332, the user management unit 333, and the device management unit 334 provided in the server 1 are additionally stored in the memory 210. In addition, the user position / direction detection program 336, the virtual object attribute information 239, the user authentication information 240, and the device management information 241 provided in the server 1 are additionally stored in the storage 211. The role of each block is as explained in FIG. 3B of the first embodiment.

[0103] FIG. 14 is a diagram showing an operation sequence in the second embodiment. In this operation sequence, the terminal 3c also serves as the server 1 in the first embodiment. Therefore, the communication between the terminal 3a and the server 1 that existed in the first embodiment (FIG. 5) is omitted. That is, when a personalization operation (S104) or a sharing operation (S108) is performed in the terminal 3c, the terminal 3c updates the virtual object attribute information 239 stored therein (S105, S109) and corrects the display state of the virtual object that it displays (S106, S110). The updated virtual object data is also transmitted to the terminal 3b. Other operations are the same as those in the first embodiment (FIG. 5).

[0104] Note that Figure 14 shows a case where a personalization action or a sharing action is performed on terminal 3c, but if a personalization action or a sharing action is performed on terminal 3b, the detection information of the personalization action or the sharing action is sent to terminal 3c, and the virtual object attribute information 239 stored in terminal 3c is updated in the same manner as described above.

[0105] According to the second embodiment, if the mixed reality display terminal is connected to a network, a plurality of users can share the same mixed reality space, and cooperative operations on virtual objects can be easily performed.

[0106] In the above embodiment, one mixed reality display terminal 3c plays the role of the server, but it is also possible to configure all mixed reality display terminals to each play the role of the server and share virtual object attribute information.

[0107] The present invention is not limited to the above-described embodiments and includes various modifications. The above-described embodiments are detailed descriptions of the entire system to clearly explain the present invention, and are not necessarily limited to those having all of the described configurations. Furthermore, it is possible to add, delete, or replace part of the configuration of each embodiment with other configurations.

[0108] Furthermore, the above-described configurations, functions, processing units, processing means, etc. may be partially or entirely implemented in hardware, for example, by designing them as integrated circuits. The above-described configurations, functions, etc. may also be implemented in software, with a processor interpreting and executing a program that implements each function. Information such as the programs, tables, and files that implement each function can be stored in a memory, a recording device such as a hard disk or SSD (Solid State Drive), or a recording medium such as an IC card, SD card, or DVD.

[0109] In addition, the control lines and information lines shown are those that are considered necessary for the explanation, and do not necessarily show all the control lines and information lines in the product. In reality, it can be assumed that almost all components are interconnected. [Explanation of symbols]

[0110] 1: Server, 2: Network, 3a, 3b: Mixed reality display terminal (terminal), 3c: Mixed reality display terminal (with built-in server function), 4a, 4b: User, 7,8,501,502,601,602:Virtual objects, 10,10': Mixed reality display system, 200: distance sensor, 203, 231: Communication I / F, 204: In-camera, 205: Out camera, 206, 242: Video output unit, 207, 243: Audio output section, 208: Input means, 209,233: Control unit, 210,234: Memory, 221,235: Storage, 239: Virtual object attribute information, 332: Virtual object management unit.

Claims

1. A mixed reality display terminal constituting a mixed reality display system in which a server and a plurality of mixed reality display terminals used by a plurality of users are connected via a network and which displays a composite of virtual objects in real space, A control unit; a communication interface for communicating with the server via the network, In the mixed reality display system, The virtual objects include shared virtual objects for which operation authority is given based on attribute information, and personal virtual objects for which operation authority is given only to a specific mixed reality display terminal based on attribute information, the control unit generates a switching operation instruction for switching between the shared virtual object and the personal virtual object based on a user input; the communication interface transmits to the server information relating to the switching operation instruction for attribute information of a virtual object required for displaying and operating the virtual object in each mixed reality display terminal, and receives attribute information updated based on the information relating to the switching operation instruction from the server; The mixed reality display terminal of the mixed reality display system, wherein the control unit performs control based on attribute information received by the communication interface from the server.

2. The mixed reality display terminal according to claim 1, A mixed reality display terminal of a mixed reality display system characterized in that it determines whether the switching operation instruction is a personalization instruction to copy the shared virtual object to generate the personal virtual object, or a sharing instruction to merge the personal virtual object with the shared virtual object from which it was copied.

3. The mixed reality display terminal according to claim 1, A mixed reality display terminal of a mixed reality display system, characterized in that among the multiple mixed reality display terminals, a mixed reality display terminal that has received the virtual object switching operation from the user and a mixed reality display terminal that has not received the virtual object switching operation from the user display different virtual object attribute information for the same virtual object displayed on each mixed reality display terminal, and displays the same virtual object in a different color, brightness, shape, or sound.

4. The mixed reality display terminal according to claim 3, A mixed reality display terminal of a mixed reality display system, characterized in that during the process of the user's switching operation detected by the operation detection unit, each of the multiple mixed reality display terminals gradually changes the color, brightness, shape, or sound displayed for the same virtual object.

5. The mixed reality display terminal according to claim 3, A mixed reality display terminal of a mixed reality display system characterized in that the same virtual object is displayed at different positions depending on the distance between the virtual object and the user in a mixed reality display terminal that has received the virtual object switching operation from the user and a mixed reality display terminal that has not received the virtual object switching operation from the user.

6. The mixed reality display terminal according to claim 3, each of the plurality of mixed reality display terminals includes a gaze detection unit that detects a gaze direction of the user; The mixed reality display terminal of the mixed reality display system is characterized in that, after the gaze detection unit detects that the user's gaze is placed on the virtual object, the display state for the virtual object is switched in accordance with the virtual object attribute information.

7. The mixed reality display terminal according to claim 3, each of the plurality of mixed reality display terminals includes a distance detection unit that detects a distance to the virtual object; The mixed reality display terminal of the mixed reality display system is characterized in that, after the distance detection unit detects that the mixed reality display terminal has approached a predetermined distance from the virtual object, the virtual object is displayed in a different color, brightness, shape, or sound according to the virtual object attribute information.

8. The mixed reality display terminal according to claim 3, each of the plurality of mixed reality display terminals includes a speed detection unit that detects a speed or an acceleration of approaching the virtual object; In the mixed reality display terminal of the mixed reality display system, after the speed detection unit detects that the mixed reality display terminal has approached the virtual object at a distance exceeding a predetermined speed or acceleration, the mixed reality display terminal displays the virtual object in a different color, brightness, shape, or sound according to the virtual object attribute information.

9. The mixed reality display terminal according to claim 2, A mixed reality display terminal of a mixed reality display system, characterized in that when the action detection unit determines that the user's action is the sharing action, the server, when merging the personal virtual object into the shared virtual object, confirms with the user whether or not to combine the parts of the shapes of the virtual objects that do not encompass each other into a new shared virtual object.

Images