Information processing device, display system, display method, and display program

Abstract

The present invention achieves smooth communication when a plurality of users communicate with each other while viewing the same three-dimensional model in a virtual space. This information processing device is provided with: an authority setting unit that sets operation authority for operating a three-dimensional model in a virtual space to any user; an authorized person image generation unit that, when the operation authority is set to a first user, generates, on the basis of the first user's operation instruction, an authorized person screen including an authorized person image obtained by operating a three-dimensional model in a virtual space based on the first user's viewpoint; a non-authorized person image generation unit that generates a non-authorized person screen including the authorized person image and a non-authorized person image that indicates a virtual space based on the viewpoint of a second user to whom the operation authority has not been set; and a control unit that performs control to display the authorized person screen on a first HMD worn by the first user, and display the non-authorized person screen on a second HMD worn by the second user.

Description

Information processing device, display system, display method, and display program

[0001] The present disclosure relates to an information processing device, a display system, a display method, and a display program.

[0002] A display system is known in which multiple users wear head-mounted displays (HMDs) to communicate with each other while viewing the same three-dimensional model in a virtual space in virtual reality or augmented reality. With this display system, each user can communicate while sharing the same three-dimensional model even when the other users are far away and cannot share the actual object.

[0003] JP 2014-206990 A JP 2022-87917 A

[0004] On the other hand, in the above-mentioned display system, multiple users wearing HMDs view the same 3D model from their own viewpoints, which makes it difficult for multiple users to communicate smoothly if they view different positions.

[0005] In addition, in the above display system, multiple users use their own controllers to operate the same 3D model in the virtual space. Therefore, if the operations of the users conflict with each other, each user cannot perform the operation they want, which hinders smooth communication.

[0006] The present disclosure aims to achieve smooth communication when multiple users communicate while viewing the same three-dimensional model in a virtual space.

[0007] An information processing device according to a first aspect of the present disclosure includes: an authority setting unit that sets operation authority for operating a three-dimensional model in a virtual space to any user; an authority image generation unit that, when the operation authority is set to a first user, generates an authorized person image that operates a three-dimensional model in the virtual space based on the viewpoint of the first user based on an operation instruction from the first user, and generates an authorized person screen including the authorized person image; an unauthorized person image generation unit that generates an unauthorized person image showing the virtual space based on the viewpoint of a second user to whom the operation authority is not set, and a unauthorized person screen including the authorized person image; and a control unit that controls the authorized person screen to be displayed on a first HMD worn by the first user, and the unauthorized person screen to be displayed on a second HMD worn by the second user.

[0008] A second aspect of the present disclosure is an information processing device according to the first aspect, wherein the authorized person image generation unit, when the operation authority is set to a second user, generates an authorized person image by operating a three-dimensional model in a virtual space based on the viewpoint of the second user based on an operation instruction from the second user; the non-authorized person image generation unit generates a non-authorized person image showing the virtual space based on the viewpoint of a first user to whom the operation authority is not set; and the control unit controls to display the authorized person screen on a second HMD worn by the second user and to display the non-authorized person screen on a first HMD worn by the first user.

[0009] A third aspect of the present disclosure is an information processing device according to the first or second aspect, wherein when the operation authority is set to a first user, the control unit controls to notify the authorized person image generation unit of viewpoint data received from the first HMD and operation instructions from the first user, and the authorized person image generation unit generates an image of a virtual space based on the viewpoint of the first user based on the viewpoint data received from the first HMD, and generates an image of a three-dimensional model in the virtual space based on the viewpoint of the first user based on the operation instructions from the first user, thereby generating the authorized person image.

[0010] A fourth aspect of the present disclosure is the information processing device according to the second aspect, wherein when the operation authority is set to a second user, the control unit controls to notify the authorized person image generation unit of viewpoint data received from the second HMD and operation instructions from the second user, and the authorized person image generation unit generates the authorized person image by generating an image of a virtual space based on the viewpoint of the second user based on the viewpoint data received from the second HMD, and generating an image of a three-dimensional model in the virtual space based on the viewpoint of the second user based on the operation instructions from the second user.

[0011] A fifth aspect of the present disclosure is an information processing device according to the third aspect, wherein the control unit, when the operation authority is set to a first user, controls to notify the authorized person image generation unit of a position specification instruction by the second user, and the authorized person image generation unit generates the authorized person image including an indicator indicating a position in a virtual space based on the viewpoint of the first user based on the position specification instruction by the second user.

[0012] A sixth aspect of the present disclosure is an information processing device according to the fourth aspect, wherein the control unit, when the operating authority is set to a second user, controls to notify the authorized person image generation unit of a position specification instruction by the first user, and the authorized person image generation unit generates the authorized person image including an indicator indicating a position in a virtual space based on the viewpoint of the second user based on the position specification instruction by the first user.

[0013] A seventh aspect of the present disclosure is an information processing device described in any one of the first to sixth aspects, further comprising a transceiver unit that transmits the voice of the first user to the second HMD and transmits the voice of the second user to the first HMD.

[0014] An eighth aspect of the present disclosure is an information processing device described in any of the first to seventh aspects, wherein the unauthorized person image generation unit generates an unauthorized person screen including the authorized person image when the second user, to whom the operating authority is not set, does not wear the second HMD, and the control unit controls to display the authorized person screen on a first HMD worn by the first user and to display the unauthorized person screen on a display device viewed by the second user.

[0015] A ninth aspect of the present disclosure is an information processing device according to the second aspect, wherein the unauthorized person image generation unit generates an unauthorized person screen including the authorized person image when the first user, to whom the operating authority is not set, does not wear the first HMD, and the control unit controls the authorized person screen to be displayed on a second HMD worn by the second user, and the unauthorized person screen to be displayed on a display device viewed by the first user.

[0016] A display system according to a tenth aspect of the present disclosure includes the information processing device according to the first or second aspect, and the first HMD and the second HMD according to the first or second aspect.

[0017] A display system according to an eleventh aspect of the present disclosure includes the information processing device according to the eighth aspect, and the first HMD and the display device according to the eighth aspect.

[0018] A display system according to a twelfth aspect of the present disclosure includes the information processing device according to the ninth aspect, and the second HMD and the display device according to the ninth aspect.

[0019] A display method according to a thirteenth aspect of the present disclosure includes the steps of: setting operation authority for operating a three-dimensional model in a virtual space to any one of users; when the operation authority is set to a first user, generating an authorized user screen including an authorized user image that operates a three-dimensional model in the virtual space based on the viewpoint of the first user based on an operation instruction from the first user; generating an unauthorized user screen including an unauthorized user image showing a virtual space based on the viewpoint of a second user to whom the operation authority is not set, and the authorized user image; and controlling the authorized user screen to be displayed on a first HMD worn by the first user, and the unauthorized user screen to be displayed on a second HMD worn by the second user, performed by a computer.

[0020] A display program according to a fourteenth aspect of the present disclosure causes a computer to execute the following steps: setting operation authority for operating a three-dimensional model in a virtual space to any one of the users; when the operation authority is set to a first user, generating an authorized user screen including an authorized user image that operates a three-dimensional model in the virtual space based on the viewpoint of the first user based on an operation instruction from the first user; generating a non-authorized user screen including an unauthorized user image showing the virtual space based on the viewpoint of a second user to whom the operation authority is not set, and the authorized user image; and controlling the authorized user screen to be displayed on a first HMD worn by the first user, and the non-authorized user screen to be displayed on a second HMD worn by the second user.

[0021] According to the present disclosure, smooth communication can be achieved when multiple users communicate while viewing the same three-dimensional model in a virtual space.

[0022] FIG. 1 is a diagram illustrating an example of a system configuration of a display system according to a first embodiment. FIG. 2 is a first diagram illustrating an example of a communication scene when a display system according to a comparative example is used. FIG. 3 is a second diagram illustrating an example of a communication scene when a display system according to a comparative example is used. FIG. 4A is a first diagram illustrating an example of a hardware configuration of an information processing device. FIG. 4B is a diagram illustrating an example of the hardware configuration of a first HMD and a second HMD. FIG. 5 is a first diagram illustrating an example of a functional configuration of an information processing device. FIG. 6 is a diagram illustrating a specific example of processing by an authorized person image generating unit. FIG. 7 is a diagram illustrating a specific example of processing by an unauthorized person image generating unit. FIG. 8A is a first diagram illustrating a specific example of processing by a control unit. FIG. 8B is a second diagram illustrating a specific example of processing by a control unit. FIG. 9 is a first flowchart illustrating the flow of virtual space display processing by an information processing device. FIG. 10 is a flowchart illustrating the flow of audio transmission and reception processing by an information processing device. FIG. 11 is a first diagram illustrating an example of a communication scene when the display system according to the first embodiment is used. FIG. 12 is a second diagram showing an example of a communication scene when the display system according to the first embodiment is used. FIG. 13 is a diagram showing an example of a system configuration of the display system according to the second embodiment. FIG. 14A is a second diagram showing an example of a hardware configuration of an information processing device. FIG. 14B is a diagram showing an example of a hardware configuration of an information terminal. FIG. 15 is a second diagram showing an example of a functional configuration of an information processing device. FIG. 16 is a third diagram showing a specific example of processing by a control unit. FIG. 17 is a second flowchart showing the flow of virtual space display processing by an information processing device. FIG. 18 is a first diagram showing an example of a communication scene when the display system according to the second embodiment is used. FIG. 19 is a second diagram showing an example of a communication scene when the display system according to the second embodiment is used.

[0023] Hereinafter, each embodiment will be described with reference to the accompanying drawings. In this specification and drawings, components having substantially the same functional configuration are designated by the same reference numerals, and redundant description will be omitted.

[0024] [First embodiment] <System configuration of display system> First, the system configuration of a display system according to the first embodiment will be described. Fig. 1 is a diagram showing an example of the system configuration of the display system according to the first embodiment. As shown in Fig. 1, the display system 100 has: an information processing device 110; a first HMD (Head Mounted Display) 121, an audio output device 122, an audio input device 123, and a controller 124; and a second HMD (Head Mounted Display) 131, an audio output device 132, an audio input device 133, and a controller 134.

[0025] In the display system 100, the information processing device 110 and the first HMD 121 are communicatively connected. In the display system 100, the information processing device 110 and the second HMD 131 are communicatively connected. According to the display system 100, the first user 120 and the second user 130 can communicate with each other while viewing the same three-dimensional model in a virtual space.

[0026] The information processing device 110 communicates with the first HMD 121 to generate an image of a virtual space (e.g., a virtual space image 140) based on the viewpoint of a first user 120 wearing the first HMD 121. The information processing device 110 communicates with the first HMD 121 to receive an operation instruction for a three-dimensional model 141 included in the virtual space image 140.

[0027] Similarly, the information processing device 110 communicates with the second HMD 131 to generate an image of the virtual space (e.g., a virtual space image 150) based on the viewpoint of the second user 130 wearing the second HMD 131. The information processing device 110 communicates with the second HMD 131 to receive an operation instruction for a three-dimensional model 151 included in the virtual space image 150.

[0028] The information processing device 110 communicates with the first HMD 121 to receive a position designation instruction that designates a position within the virtual space image (virtual space image 140 or 150).

[0029] Similarly, the information processing device 110 communicates with the second HMD 131 to receive a position designation instruction that designates a position within the virtual space image (virtual space image 140 or 150).

[0030] Note that the operation authority to operate the three-dimensional model in the virtual space is granted to either the first user 120 or the second user 130. The information processing device 110 is set with the fact that the operation authority has been granted to the first user 120 or the fact that the operation authority has been granted to the second user 130. The information processing device 110 switches processing depending on whether the operation authority has been granted to the first user 120 or the second user 130.

[0031] When the operation authority is set to the first user 120, the information processing device 110: - operates the three-dimensional model 141 in the generated virtual space image 140 in accordance with the operation instruction from the first user 120, and - generates a virtual space image in which an indicator (not shown in Figure 1) has been moved to a position in accordance with the position specification instruction from the first user 120 or the second user 130, and sets this as an authorized person image.

[0032] Furthermore, if the operation authority is not set for the first user 120, the information processing device 110 sets the generated virtual space image 140 as an unauthorized person image.

[0033] When the operation authority is set to the second user 130, the information processing device 110: - operates the three-dimensional model 151 in the generated virtual space image 150 in accordance with an operation instruction from the second user 130, and - generates a virtual space image in which an indicator (not shown in FIG. 1 ) is moved to a position in accordance with a position specification instruction from the first user 120 or the second user 130, and sets the generated virtual space image 150 as an authorized person image. Furthermore, when the operation authority is not set to the second user 130, the information processing device 110: - sets the generated virtual space image 150 as an unauthorized person image.

[0034] Furthermore, when the operation authority is set to the first user 120, the information processing device 110 transmits to the first HMD 121 an authorized person screen including an authorized person image generated based on the virtual space image 140. When the operation authority is not set to the first user, the information processing device 110 transmits to the first HMD 121 an unauthorized person screen including an unauthorized person image generated based on the virtual space image 140 and an authorized person image generated based on the virtual space image 150.

[0035] When the operation authority is set to the second user 130, the information processing device 110 transmits to the second HMD 131 an authorized person screen including an authorized person image generated based on the virtual space image 150. When the operation authority is not set to the second user, the information processing device 110 transmits to the second HMD 131 an unauthorized person screen including an unauthorized person image generated based on the virtual space image 150 and an authorized person image generated based on the virtual space image 140.

[0036] Furthermore, the information processing device 110 communicates with the first HMD 121 and the second HMD 131 to transmit and receive audio data so that the audio of the first user 120 acquired by the audio input device 123 is output on the audio output device 132. The information processing device 110 communicates with the first HMD 121 and the second HMD 131 to transmit and receive audio data so that the audio of the second user 130 acquired by the audio input device 133 is output on the audio output device 122.

[0037] The first HMD 121 is an HMD worn by the first user 120. The first HMD 121 has various sensors such as an acceleration sensor, a gyro sensor, and a geomagnetic sensor, and acquires viewpoint data of the first user 120 by measuring the head position, head orientation, head rotation direction, etc. of the first user 120. The first HMD 121 transmits the acquired viewpoint data to the information processing device 110.

[0038] The first HMD 121 communicates with the audio input device 123 and the audio output device 122. The first HMD 121 acquires audio data from the audio input device 123 and transmits it to the information processing device 110, and also receives audio data from the information processing device 110 and notifies the audio output device 122 of the audio data.

[0039] The first HMD 121 communicates with the controller 124. The first HMD 121 acquires an operation instruction or a position designation instruction from the controller 124 and transmits it to the information processing device 110.

[0040] The first HMD 121 has a display device, and displays the authorized user screen or unauthorized user screen transmitted from the information processing device 110 on the display device.

[0041] The audio output device 122 outputs audio data notified from the first HMD 121. The audio input device 123 acquires the audio of the first user 120 and notifies the first HMD 121. The controller 124 has various sensors such as an acceleration sensor, a gyro sensor, and a geomagnetic sensor, and measures various sensor data such as the hand position, hand orientation, and hand movement direction of the first user 120, and acquires operation instructions or position designation instructions from the first user 120. The controller 124 notifies the first HMD 121 of the acquired operation instructions or position designation instructions.

[0042] When the operation authority is set to the first user 120, the controller 124 notifies the first HMD 121 of either an operation instruction or a position designation instruction by operating a selector switch (not shown). When the operation authority is not set to the first user 120, the controller 124 notifies the first HMD 121 of a position designation instruction regardless of whether the selector switch (not shown) is operated.

[0043] The second HMD 131 is an HMD worn by the second user 130. The second HMD 131 has various sensors such as an acceleration sensor, a gyro sensor, and a geomagnetic sensor, and acquires viewpoint data of the second user 130 by measuring the head position, head orientation, head rotation direction, etc. of the second user 130. The second HMD 131 transmits the acquired viewpoint data to the information processing device 110.

[0044] The second HMD 131 communicates with the audio input device 133 and the audio output device 132. The second HMD 131 acquires audio data from the audio input device 133 and transmits it to the information processing device 110, and also receives audio data from the information processing device 110 and notifies the audio output device 132 of the audio data.

[0045] The second HMD 131 communicates with the controller 134. The second HMD 131 acquires an operation instruction or a position designation instruction from the controller 134 and transmits it to the information processing device 110.

[0046] The second HMD 131 has a display device, and displays the authorized user screen or unauthorized user screen transmitted from the information processing device 110 on the display device.

[0047] The audio output device 132 outputs audio data notified from the second HMD 131. The audio input device 133 acquires the audio of the second user 130 and notifies the second HMD 131. The controller 134 has various sensors such as an acceleration sensor, a gyro sensor, and a geomagnetic sensor, and measures various sensor data such as the hand position, hand orientation, and hand movement direction of the second user 130, and acquires operation instructions or position designation instructions from the second user 130. The controller 134 notifies the second HMD 131 of the acquired operation instructions or position designation instructions.

[0048] When the operation authority is set to the second user 130, the controller 134 notifies the second HMD 131 of either an operation instruction or a position designation instruction by operating a selector switch (not shown). When the operation authority is not set to the second user 130, the controller 134 notifies the second HMD 131 of a position designation instruction regardless of whether the selector switch (not shown) is operated.

[0049] In the above description, it is assumed that the first user 120 and the second user 130 are in a position where they cannot have a direct conversation with each other, and that audio data is transmitted and received via the information processing device 110. However, for example, if the first user 120 and the second user 130 are in the same room and can have a direct conversation with each other, the audio output device 122, the audio input device 123, the audio output device 132, and the audio input device 133 are not essential components.

[0050] <Communication scene when using the display system of the comparative example> Next, we will explain the problems that arise when a first user 120 and a second user 130 communicate while viewing the same three-dimensional model in a virtual space using the display system of the comparative example.

[0051] FIG. 2 is a first diagram showing an example of a communication scene when a display system of the comparative example is used.

[0052] 2 , a virtual space image 210 is an example of a virtual space image displayed on a display device of the first HMD 121 in the display system of the comparative example. In the case of the display system of the comparative example, the display device of the first HMD 121 displays a virtual space image 210 based on the viewpoint of the first user 120, in which an indicator 211 has been moved to a position corresponding to a position designation instruction transmitted from the first HMD 121.

[0053] 2, a virtual space image 220 is an example of a virtual space image displayed on a display device of the second HMD 131 in the display system of the comparative example. As shown in FIG. 2, in the display system of the comparative example, the virtual space image 220 based on the viewpoint of the second user 130 is displayed on the display device of the second HMD 131.

[0054] In this display situation, suppose that first user 120 utters, for example, "I want to change the material of this surface" regarding a specific surface of three-dimensional model 141 pointed to by indicator 211. In this case, second user 130 cannot grasp which surface of three-dimensional model 151 first user 120 is referring to as the surface whose material he or she wishes to change, and therefore cannot communicate smoothly.

[0055] This is because the virtual space image 220 viewed by the second user 130 is a virtual space image based on the viewpoint of the second user 130, and the second user 130 is unable to view the three-dimensional model 141 on which a specific surface is pointed out by the first user 120. Furthermore, the virtual space image 220 viewed by the second user 130 does not display the indicator 211 pointed out by the first user 120. This is because the second user 130 is unable to recognize which surface of the three-dimensional model 151 the surface pointed out by the first user 120 corresponds to.

[0056] FIG. 3 is a second diagram showing an example of a communication scene when the display system of the comparative example is used.

[0057] 3 , virtual space image 310 is an example of a virtual space image displayed on a display device of first HMD 121 in a display system of the comparative example. As shown in FIG. 3 , in the display system of the comparative example, virtual space image 310 based on the viewpoint of first user 120 is displayed on the display device of first HMD 121. Therefore, for example, when first user 120 attempts to view the window side surface of three-dimensional model 141 in the virtual space, first user 120 moves controller 124 in a counterclockwise arc to rotate three-dimensional model 141 by 90 degrees counterclockwise.

[0058] 3 , virtual space image 320 is an example of a virtual space image displayed on a display device of second HMD 131 in a display system of the comparative example. As shown in FIG. 3 , in the display system of the comparative example, virtual space image 320 based on the viewpoint of second user 130 is displayed on the display device of second HMD 131. Therefore, for example, when second user 130 attempts to view the window side surface of three-dimensional model 151 in the virtual space, second user 130 moves controller 134 in an arc from the back to the front, thereby rotating three-dimensional model 151 vertically by 180 degrees.

[0059] In this way, in a situation where the operations of the first user 120 and the second user 130 conflict with each other, neither the first user 120 nor the second user 130 can move the three-dimensional model 141 or the three-dimensional model 151 in the desired direction. As a result, neither the first user 120 nor the second user 130 can see the surface they are trying to view, and communication cannot be carried out smoothly. This is because, when multiple users simultaneously give different operation instructions to the same three-dimensional model, the same three-dimensional model will move in a direction that is the sum of the operation instructions.

[0060] In this way, when a first user 120 and a second user 130 communicate while viewing the same three-dimensional model in a virtual space using the display system of the comparative example, problems arise in which smooth communication cannot be achieved in the following situations: - a scene in which one user gives a position specification instruction, or - a scene in which multiple users give operation instructions simultaneously.

[0061] The display system 100 according to the first embodiment aims to solve such problems and realize smooth communication when multiple users communicate while viewing the same three-dimensional model in a virtual space. Details of the display system 100 according to the first embodiment will be described below.

[0062] <Hardware Configuration of Information Processing Device> First, the hardware configuration of the information processing device 110 will be described. Fig. 4A is a first diagram showing an example of the hardware configuration of the information processing device. As shown in Fig. 4A, the information processing device 110 has a processor 401, a memory 402, an auxiliary storage device 403, an input / output device 404, a communication device 405, and a drive device 406. The hardware components of the information processing device 110 are connected to each other via a bus 407.

[0063] The processor 401 has various computing devices such as a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit). The processor 401 reads and executes various programs on the memory 402. The various programs here include, for example, a display program, which will be described later.

[0064] The memory 402 has a main storage device such as a read-only memory (ROM) or a random access memory (RAM). The processor 401 and the memory 402 form a so-called computer. The computer realizes each functional unit by the processor 401 executing various programs read onto the memory 402.

[0065] The auxiliary storage device 403 stores various programs and various data used when the processor 401 executes the programs.

[0066] The input / output device 404 is a connection device that connects to an operation device 411, which is an example of a user interface device, and a display device 412. The communication device 405 is a communication device for communicating with the first HMD 121 and the second HMD 131 via a network.

[0067] The drive device 406 is a device for loading a recording medium 420. The recording medium 420 here includes media that optically, electrically, or magnetically record information, such as CD-ROMs, flexible disks, and magneto-optical disks. The recording medium 420 may also include media such as semiconductor memories that electrically record information, such as ROMs and flash memories.

[0068] The various programs to be installed in the auxiliary storage device 403 are installed, for example, by setting the distributed recording medium 420 in the drive device 406 and reading out the various programs recorded on the recording medium 420. Alternatively, the various programs to be installed in the auxiliary storage device 403 may be installed by being downloaded from a network via the communication device 405.

[0069] <Hardware Configuration of First HMD and Second HMD> Next, the hardware configuration of the first HMD 121 and the second HMD will be described. Fig. 4B is a diagram showing an example of the hardware configuration of the first HMD and the second HMD. As shown in (a) of Fig. 4B, the first HMD 121 has a processor 431, a memory 432, an auxiliary storage device 433, a sensor 434, a display device 435, a wireless device 436, and a communication device 437. The hardware components of the first HMD 121 are connected to each other via a bus 438.

[0070] The processor 431 has various arithmetic devices such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), etc. The processor 431 reads various programs into the memory 432 and executes them.

[0071] The memory 432 has a main storage device such as a read-only memory (ROM) or a random access memory (RAM). The processor 431 and the memory 432 form a so-called computer. The computer realizes each functional unit by the processor 431 executing various programs read onto the memory 432.

[0072] The auxiliary storage device 433 stores various programs and various data used when the various programs are executed by the processor 431 .

[0073] The sensor 434 includes various sensors such as the acceleration sensor, gyro sensor, and geomagnetic sensor described above, and acquires viewpoint data of the first user 120. The display device 435 displays the authorized person screen or unauthorized person screen described above.

[0074] The wireless device 436 is a device for transmitting and receiving voice data, operation instructions or position designation instructions between the voice output device 122 , the voice input device 123 , the controller 124 and the first HMD 121 .

[0075] The communication device 437 is a device for transmitting and receiving viewpoint data, transmitting and receiving authorized user screens or unauthorized user screens, transmitting and receiving audio data, and transmitting and receiving operation instructions or position specification instructions between the information processing device 110 and the first HMD 121.

[0076] The various programs installed in the auxiliary storage device 433 are installed by being downloaded from a network via the communication device 437, for example.

[0077] 4B(b), the second HMD 131 has a processor 441, a memory 442, an auxiliary storage device 443, a sensor 444, a display device 445, a wireless device 446, and a communication device 447. The hardware components of the second HMD 131 are connected to each other via a bus 448. Note that the hardware components of the second HMD 131 are similar to the hardware components of the first HMD 121, and therefore will not be described here.

[0078] <Functional Configuration of Information Processing Device> Next, the functional configuration of the information processing device 110 will be described. Fig. 5 is a first diagram showing an example of the functional configuration of the information processing device. As described above, a display program is installed in the information processing device 110, and by executing this program, the information processing device 110 realizes each of the functional units shown in Fig. 5.

[0079] 5 , the functional units realized in the information processing device 110 include a virtual space image providing unit 501, an authorized person image generating unit 511, and an unauthorized person image generating unit 512. The functional units realized in the information processing device 110 also include an authorized person viewpoint data acquiring unit 521, an authorized person instruction accepting unit 522, an unauthorized person instruction accepting unit 523, and an unauthorized person viewpoint data acquiring unit 524. The functional units realized in the information processing device 110 also include an authority setting unit 531, a control unit 532, and an audio data transmitting / receiving unit 540.

[0080] The virtual space image providing unit 501 provides a base virtual space image to the authorized person image generating unit 511 and the unauthorized person image generating unit 512. The virtual space image provided by the virtual space image providing unit 501 includes a three-dimensional model.

[0081] The authorized person image generating unit 511 acquires viewpoint data of a user for whom operation authority has been set from the authorized person viewpoint data acquiring unit 521. The authorized person image generating unit 511 generates a virtual space image based on the acquired viewpoint data from the virtual space image provided by the virtual space image providing unit 501.

[0082] The authorized person image generating unit 511 acquires an operation instruction from a user who has been granted operation authority from the authorized person instruction receiving unit 522. The authorized person image generating unit 511 operates a three-dimensional model included in a virtual space image based on viewpoint data in accordance with the operation instruction, thereby generating a virtual space image after the three-dimensional model has operated.

[0083] The authorized person image generating unit 511 acquires a position designation instruction from a user who is not granted operation authority from the unauthorized person instruction receiving unit 523. The authorized person image generating unit 511 generates a virtual space image in which the indicator has been moved to a position corresponding to the position designation instruction in the virtual space image after the three-dimensional model has been operated.

[0084] The authorized person image generating unit 511 notifies the unauthorized person image generating unit 512 of the virtual space image in which the indicator has been moved as an authorized person image. The authorized person image generating unit 511 also generates an authorized person screen including the virtual space image in which the indicator has been moved, and notifies the control unit 532 of the generated screen.

[0085] The authority person viewpoint data acquisition unit 521 acquires viewpoint data of a user who has been granted operation authority, and notifies the authority person image generation unit 511 of the data.

[0086] The authorized person instruction receiving unit 522 acquires an operation instruction from a user who has been granted operation authority, and notifies the authorized person image generating unit 511 of the operation instruction.

[0087] The unauthorized person instruction receiving unit 523 acquires a position designation instruction from a user who is not authorized to operate, and notifies the authorized person image generating unit 511 of the instruction.

[0088] The unauthorized person image generation unit 512 acquires viewpoint data of a user for whom operation authority has not been set from the unauthorized person viewpoint data acquisition unit 524. The unauthorized person image generation unit 512 generates a virtual space image based on the acquired viewpoint data from the virtual space image provided by the virtual space image provision unit 501, and sets it as an unauthorized person image. The unauthorized person image generation unit 512 also acquires an authorized person image from the authorized person image generation unit 511. The unauthorized person image generation unit 512 generates an unauthorized person screen including an authorized person image and an unauthorized person image, and notifies the control unit 532.

[0089] The unauthorized person viewpoint data acquisition unit 524 acquires viewpoint data of a user to whom operation authority has not been set, and notifies the unauthorized person image generation unit 512 of the data.

[0090] The authority setting unit 531 sets operation authority for operating a three-dimensional model in the virtual space to either the first user 120 or the second user 130. The authority setting unit 531 notifies the control unit 532 of the user for whom the operation authority has been set.

[0091] The control unit 532 switches the notification or transmission destination of viewpoint data, authorized user screen or unauthorized user screen, operation instructions or position specification instructions sent and received between the first HMD 121 and the second HMD 131 according to the set operation authority.

[0092] The voice data transmission / reception unit 540 transmits and receives voice data between the first HMD 121 and the second HMD.

[0093] <Details of Processing by Each Functional Unit of Information Processing Apparatus> Next, details of processing by each functional unit of the information processing apparatus 110 (here, the authorized person image generating unit 511, the unauthorized person image generating unit 512, and the control unit 532) will be described.

[0094] (1) Details of Processing by Authorized Person Image Generation Unit 511 First, details of processing by the authorized person image generation unit 511 will be described. Fig. 6 is a diagram showing a specific example of processing by the authorized person image generation unit. As shown in Fig. 6, the authorized person image generation unit 511 further includes a virtual space image acquisition unit 601, a viewpoint change unit 602, a model control unit 603, a pointer control unit 604, and an authorized person screen output unit 605.

[0095] The virtual space image acquisition unit 601 acquires the virtual space image provided by the virtual space image provision unit 501 and notifies the viewpoint change unit 602 of the acquired image.

[0096] The viewpoint changing unit 602 acquires viewpoint data of a user for whom operation authority has been set from the authorized user viewpoint data acquisition unit 521. Here, the description will be made assuming that the operation authority has been set to the first user 120. The viewpoint changing unit 602 generates a virtual space image 610 based on the viewpoint data of the first user 120. Note that the virtual space image 610 includes, in addition to the three-dimensional model 141, an indicator 611 that moves in accordance with an operation instruction or a position designation instruction from the first user 120, and an indicator 612 that moves in accordance with a position designation instruction from the second user 130.

[0097] The model control unit 603 acquires an operation instruction from the first user 120 from the authority instruction receiving unit 522. The model control unit 603 operates the three-dimensional model 141 included in the virtual space image 610 in accordance with the operation instruction from the first user 120, thereby generating a virtual space image 620 after the operation of the three-dimensional model 141.

[0098] The indicator control unit 604 acquires a position specification instruction from the second user 130 from the unauthorized person instruction receiving unit 523. The indicator control unit 604 moves the indicator 612 included in the virtual space image 620 in accordance with the position specification instruction from the second user 130, thereby generating a virtual space image in which the indicator 612 has been moved, and sets this as an authorized person image 630. The indicator control unit 604 notifies the unauthorized person image generation unit 512 of the authorized person image 630, and also notifies the authorized person screen output unit 605 of the authorized person image 630.

[0099] The authority screen output unit 605 generates an authority screen 640 including the authority image 630 and notifies the control unit 532 of it.

[0100] (2) Details of Processing by Unauthorized Person Image Generation Unit 512 Next, details of processing by the unauthorized person image generation unit 512 will be described. Fig. 7 is a diagram showing a specific example of processing by the unauthorized person image generation unit. As shown in Fig. 7, the unauthorized person image generation unit 512 further includes a virtual space image acquisition unit 701, a viewpoint change unit 702, and an unauthorized person screen output unit 703.

[0101] The virtual space image acquisition unit 701 acquires the virtual space image provided by the virtual space image provision unit 501 and notifies the viewpoint change unit 702 of the acquired image.

[0102] The viewpoint change unit 702 acquires viewpoint data of the second user 130, for whom operation authority has not been set, from the unauthorized person viewpoint data acquisition unit 524. The viewpoint change unit 702 generates an unauthorized person image 710 based on the viewpoint data of the second user 130, and notifies the unauthorized person screen output unit 703 of the generated image.

[0103] The unauthorized person screen output unit 703 generates an unauthorized person screen 720 using the authorized person image 630 notified from the authorized person image generation unit 511 and the unauthorized person image 710 notified from the viewpoint change unit 702, and notifies the control unit 532. As shown in Fig. 7 , the unauthorized person screen 720 includes a main area and a sub area, and the authorized person image 630 is displayed in the main area, and the unauthorized person image 710 is displayed in the sub area.

[0104] (3) Details of Processing by Control Unit 532 Next, details of processing by control unit 532 will be described. FIG. 8A is a first diagram showing a specific example of processing by the control unit. As described above, control unit 532 switches the notification or transmission destination of viewpoint data, authorized user screen or unauthorized user screen, operation instructions, or position specification instructions according to the set operation authority. FIG. 8A shows how the notification or transmission destination is switched when the operation authority is set to first user 120.

[0105] Specifically, when operation authority is set to the first user 120, the control unit 532: - notifies the authorized person viewpoint data acquisition unit 521 of the viewpoint data transmitted from the first HMD 121; - notifies the authorized person instruction receiving unit 522 of the operation instruction or position designation instruction transmitted from the first HMD 121; - notifies the non-authorized person instruction receiving unit 523 of the position designation instruction transmitted from the second HMD 131; and - transmits the authorized person screen generated by the authorized person image generation unit 511 to the first HMD 121.

[0106] Furthermore, when the operating authority is set to the first user 120, the control unit 532: - notifies the viewpoint data transmitted from the second HMD 131 to the unauthorized person viewpoint data acquisition unit 524; and - transmits the unauthorized person screen generated by the unauthorized person image generation unit 512 to the second HMD 131.

[0107] 8B is a second diagram showing a specific example of processing by the control unit. As described above, the control unit 532 switches the notification or transmission destination of viewpoint data, the authorized user screen or unauthorized user screen, and operation instructions or position specification instructions depending on the set operation authority. FIG. 8B shows how the notification or transmission destination is switched when the operation authority is set to the second user 130.

[0108] Specifically, when operation authority is set to the second user 130, the control unit 532: - notifies the authorized person viewpoint data acquisition unit 521 of the viewpoint data transmitted from the second HMD 131; - notifies the authorized person instruction receiving unit 522 of the operation instruction or position designation instruction transmitted from the second HMD 131; - notifies the non-authorized person instruction receiving unit 523 of the position designation instruction transmitted from the first HMD 121; and - transmits the authorized person screen generated by the authorized person image generation unit 511 to the second HMD 131.

[0109] Also, when the operating authority is set to the second user 130, the control unit 532: - notifies the viewpoint data transmitted from the first HMD 121 to the unauthorized person viewpoint data acquisition unit 524, and - transmits the unauthorized person screen generated by the unauthorized person image generation unit 512 to the first HMD 121.

[0110] <Flow of Virtual Space Display Processing by Information Processing Device> Next, a description will be given of the flow of virtual space display processing by the information processing device 110. Fig. 9 is a first flowchart showing the flow of virtual space display processing by the information processing device.

[0111] In step S901, the information processing apparatus 110 accepts the setting of operation authority.

[0112] In step S902, the information processing device 110 switches the notification or transmission destination of viewpoint data, authorized user screen or unauthorized user screen, operation instructions or position designation instructions sent and received between the first HMD 121 and the second HMD 131 depending on the set operation authority.

[0113] In step S903, the information processing device 110 acquires a virtual space image.

[0114] In step S904, the information processing apparatus 110 acquires viewpoint data of users who have been granted operation authority, operation instructions of users who have been granted operation authority, and position designation instructions of users who have not been granted operation authority.

[0115] In step S905, the information processing device 110 generates an image of an authorized person based on the viewpoint data, operation instructions, and position designation instructions acquired in step S904.

[0116] In step S906, the information processing apparatus 110 acquires viewpoint data of users to whom operation authority has not been set.

[0117] In step S907, the information processing device 110 generates an unauthorized person image based on the viewpoint data acquired in step S906.

[0118] In step S908, the information processing apparatus 110 generates an authority screen including an authority image, and transmits the authority screen to the user for whom operation authority has been set.

[0119] In step S909, the information processing apparatus 110 generates an unauthorized person screen including an authorized person image and an unauthorized person image, and transmits the unauthorized person screen to a user who has not been granted operation authority.

[0120] In step S910, the information processing device 110 determines whether to end the virtual space display process. If it is determined in step S910 that the virtual space display process should be continued (NO in step S910), the process returns to step S903. On the other hand, if it is determined in step S910 that the virtual space display process should be ended (YES in step S910), the virtual space display process is ended.

[0121] <Flow of Audio Transmission and Reception Processing by Information Processing Device> Next, a description will be given of the flow of audio transmission and reception processing by the information processing device 110. Fig. 10 is a flowchart showing the flow of audio transmission and reception processing by the information processing device.

[0122] In step S1001, the information processing device 110 receives the voice of the first user 120 acquired by the voice input device 123 from the first HMD 121 and transmits it to the second HMD 131, thereby outputting the voice of the first user 120 from the voice output device 132.

[0123] In step S1002, the information processing device 110 receives the voice of the second user 130 acquired by the voice input device 133 from the second HMD 131 and transmits it to the first HMD 121, thereby outputting the voice of the second user 130 from the voice output device 122.

[0124] The order of the process in step S1001 and the process in step S1002 may be reversed, or the process in step S1001 and the process in step S1002 may be executed in parallel.

[0125] In step S1003, the information processing device 110 determines whether to end the audio transmission / reception process. If it is determined in step S1003 that the audio transmission / reception process should be continued (NO in step S1003), the process returns to step S1001. On the other hand, if it is determined in step S1003 that the audio transmission / reception process should be ended (YES in step S1003), the audio transmission / reception process is ended.

[0126] <Communication scene when using the display system according to the first embodiment> Next, a scene will be described in which a first user 120 and a second user 130 communicate while viewing the same three-dimensional model in a virtual space using the display system 100 according to the first embodiment.

[0127] 11 is a first diagram showing an example of a communication scene when the display system according to the first embodiment is used, and shows an example of a communication scene when operation authority is set to the first user 120. As shown in FIG. 11 , in the case of the display system 100, an authorized person screen 640 including an authorized person image 1101 is displayed on the display device 435 of the first HMD 121. Also, in the case of the display system 100, an unauthorized person screen 720 including the authorized person image 1101 and an unauthorized person image 1102 is displayed on the display device 445 of the second HMD 131. However, in the display device 445 of the second HMD 131, the unauthorized person screen 720 may not include the unauthorized person image 1102, by selection by the unauthorized person or by default setting.

[0128] Under such a display situation, suppose that the first user 120 utters, for example, "I want to change the material of this surface." In this case, the second user 130 can see the authorized person image 1101 included in the unauthorized person screen 720 and understand that the surface whose material the first user 120 wishes to change is the surface pointed to by the first user 120's pointer. As a result, smooth communication can be achieved between the first user 120 and the second user 130.

[0129] Although not shown in Figure 11, let's assume that the second user 130, while viewing the authorized person image 1101 included in the unauthorized person screen 720, moves the pointer of the second user 130 and points to a surface different from the surface pointed to by the pointer of the first user 120.

[0130] Under such a display situation, suppose that the second user 130 utters, for example, "I want to change the material of this surface." In this case, by visually checking the authority image 1101 included in the authority screen 640, the first user 120 can understand that the surface whose material the second user 130 wishes to change is the surface pointed to by the pointer of the second user 130. As a result, smooth communication can be achieved between the first user 120 and the second user 130.

[0131] Fig. 12 is a second diagram showing an example of a communication scene when the display system according to the first embodiment is used, and shows an example of a communication scene when operation authority is set to the first user 120. As shown in Fig. 12, in the case of the display system 100, operation authority is set to only one user, and multiple users do not have operation authority at the same time.

[0132] Therefore, for example, even if the first user 120 and the second user 130 try to view the window side surface of the three-dimensional model in the virtual space, only the first user 120 can operate the three-dimensional model. In other words, in the case of the display system 100 according to the first embodiment, it is possible to avoid a situation in which different operation instructions are given at the same time.

[0133] Furthermore, the first user 120 rotates the three-dimensional model 90 degrees counterclockwise by moving the controller 124 in a counterclockwise arc. This allows the first user 120 to view a desired side of the three-dimensional model in the virtual space by viewing the authorized person image 1201' included in the authorized person screen 640. Furthermore, the second user 130 can view a desired side of the three-dimensional model in the virtual space by viewing the authorized person image 1201' included in the non-authorized person screen 720.

[0134] <Summary> As is clear from the above description, the information processing device 110 according to the first embodiment: - assigns operation authority for operating a three-dimensional model in a virtual space to either the first user 120 or the second user 130. - If operation authority is assigned to the first user 120, generates an authorized user image in which the three-dimensional model in the virtual space is operated based on the viewpoint of the first user 120, based on an operation instruction from the first user 120, and generates an authorized user screen including the authorized user image. - generates an unauthorized user image showing the virtual space based on the viewpoint of the second user 130, for whom operation authority has not been assigned, and an unauthorized user screen including the authorized user image. - controls so that the authorized user screen is displayed on the first HMD 121 worn by the first user 120, and the unauthorized user screen is displayed on the second HMD 131 worn by the second user 130.

[0135] In this way, in the information processing device 110 according to the first embodiment, operation authority is set for any one user, and an authorized person image is shared among multiple users. As a result, according to the first embodiment, smooth communication can be realized when multiple users communicate while viewing the same three-dimensional model in a virtual space.

[0136] Second Embodiment In the first embodiment described above, a communication scene is described in which the first user 120 wears the first HMD 121 and the second user 130 wears the second HMD 131. In contrast, in the second embodiment, a communication scene is described in which the first user 120 wears the first HMD and the second user 130 does not wear the second HMD 131 and views the display device. Note that the description will focus on differences from the first embodiment described above, and in the second embodiment, it is assumed that operation authority is set only to the first user 120.

[0137] <System configuration of display system> First, the system configuration of the display system according to the second embodiment will be described. Fig. 13 is a diagram showing an example of the system configuration of the display system according to the second embodiment. As shown in Fig. 13, the display system 1300 includes: an information processing device 110; a first HMD 121, an audio output device 122, an audio input device 123, and a controller 124; and an information terminal 1301, an audio output device 132, an audio input device 133, and a controller 134.

[0138] In the display system 1300, the information processing device 110 and the information terminal 1301 are communicatively connected. According to the display system 1300, the first user 120 and the second user 130 can communicate with each other while visually viewing a three-dimensional model in a virtual space.

[0139] The information processing device 110 communicates with the information terminal 1301 to receive a position designation instruction that designates a position within the virtual space image 140 .

[0140] The information processing device 110 transmits an unauthorized person screen including an authorized person image generated based on the virtual space image 140 to the information terminal 1301 .

[0141] The information processing device 110 communicates with the first HMD 121 and the information terminal 1301 to transmit and receive audio data so that the audio of the first user 120 acquired by the audio input device 123 is output on the audio output device 132. The information processing device 110 communicates with the first HMD 121 and the information terminal 1301 to transmit and receive audio data so that the audio of the second user 130 acquired by the audio input device 133 is output on the audio output device 122.

[0142] The information terminal 1301 has a display device that is visually recognized by the second user 130. The information terminal 1301 displays the unauthorized user screen transmitted from the information processing device 110 on the display device.

[0143] The information terminal 1301 communicates with the audio input device 133 and the audio output device 132. The information terminal 1301 acquires audio data from the audio input device 133 and transmits it to the information processing device 110, and also receives audio data from the information processing device 110 and notifies the audio output device 132 of the audio data.

[0144] The information terminal 1301 communicates with the controller 134. The information terminal 1301 acquires a position designation instruction from the controller 134 and transmits it to the information processing device 110.

[0145] The audio output device 132 outputs audio data notified from the information terminal 1301. The audio input device 133 acquires the audio of the second user 130 and notifies the information terminal 1301. The controller 134 has various sensors such as an acceleration sensor, a gyro sensor, and a geomagnetic sensor, and measures various sensor data such as the hand position, hand orientation, and hand movement direction of the second user 130, and acquires a position designation instruction from the second user 130. The controller 134 notifies the information terminal 1301 of the acquired position designation instruction.

[0146] <Hardware configuration of information processing device> Next, the hardware configuration of the information processing device 110 will be described. Fig. 14A is a second diagram showing an example of the hardware configuration of the information processing device. The difference from the hardware configuration of the information processing device 110 shown in Fig. 4A is that in the case of the hardware configuration shown in Fig. 14A, the communication device 405 communicates with the first HMD 121 and the information terminal 1301.

[0147] <Hardware Configuration of Information Terminal> Next, the hardware configuration of the information terminal 1301 will be described. Fig. 14B is a diagram showing an example of the hardware configuration of the information terminal. As shown in Fig. 14B, the information terminal 1301 has a processor 1401, a memory 1402, an auxiliary storage device 1403, a display device 1404, a wireless device 1405, and a communication device 1406. The hardware components of the information terminal 1301 are connected to each other via a bus 1407.

[0148] The hardware components of the information terminal 1301 are generally the same as those of the second HMD 131, and therefore a description of each piece of hardware will be omitted here.

[0149] <Functional configuration of information processing device> Next, the functional configuration of the information processing device 110 will be described. Fig. 15 is a second diagram showing an example of the functional configuration of the information processing device. The differences from the functional configuration shown in Fig. 5 are that the function of the unauthorized person image generation unit 1511 is different from the function of the unauthorized person image generation unit 512 in Fig. 5, and that the function of the control unit 1521 is different from the function of the control unit 532 in Fig. 5.

[0150] The unauthorized person image generating unit 1511 acquires the authorized person image from the authorized person image generating unit 511. The unauthorized person image generating unit 1511 generates an unauthorized person screen including the acquired authorized person image, and notifies the control unit 1521 of the generated screen.

[0151] The control unit 1521 notifies or transmits to a notification destination or a transmission destination viewpoint data, an authorized person screen or an unauthorized person screen, an operation instruction or a position specification instruction transmitted and received between the first HMD 121 and the information terminal 1301.

[0152] <Details of Processing by Each Functional Unit of Information Processing Apparatus> Next, a description will be given of details of processing by the control unit 1521 of the information processing apparatus 110. Fig. 16 is a third diagram showing a specific example of processing by the control unit.

[0153] In the display system 1300 according to the second embodiment, the control unit 1521: notifies the authorized person viewpoint data acquisition unit 521 of viewpoint data transmitted from the first HMD 121; notifies the authorized person instruction receiving unit 522 of operation instructions or position designation instructions transmitted from the first HMD 121; notifies the unauthorized person instruction receiving unit 523 of position designation instructions transmitted from the information terminal 1301; transmits the authorized person screen generated by the authorized person image generation unit 511 to the first HMD 121; and transmits the unauthorized person screen generated by the unauthorized person image generation unit 512 to the information terminal 1301.

[0154] <Flow of virtual space display processing by information processing device> Next, the flow of virtual space display processing by the information processing device 110 according to the second embodiment will be described. Fig. 17 is a second flowchart showing the flow of virtual space display processing by the information processing device. The differences from the first flowchart shown in Fig. 9 are steps S1701 and S1702.

[0155] In step S1701, the information processing apparatus 110 sets the notification or transmission destination of viewpoint data, an authorized person screen or unauthorized person screen, operation instructions, or position designation instructions transmitted and received between the first HMD 121 and the information terminal 1301.

[0156] In step S1702 , the information processing apparatus 110 generates an unauthorized person screen including an authorized person image and transmits it to the information terminal 1301 .

[0157] <Communication scene when using the display system according to the second embodiment> Next, a scene will be described in which a first user 120 and a second user 130 communicate while viewing the same three-dimensional model in a virtual space using the display system 1300 according to the second embodiment.

[0158] 18 is a first diagram showing an example of communication when the display system according to the second embodiment is used. As shown in FIG. 18 , in the case of the display system 1300, an authorized person screen 640 including an authorized person image 1101 is displayed on the display device 435 of the first HMD 121. Also, in the case of the display system 1300, an unauthorized person screen 720 including the authorized person image 1101 is displayed on the display device 1404 of the information terminal 1301.

[0159] Under such a display situation, suppose that the first user 120 utters, for example, "I want to change the material of this surface." In this case, the second user 130 can see the authorized person image 1101 included in the unauthorized person screen 720 and understand that the surface whose material the first user 120 wishes to change is the surface pointed to by the first user 120's pointer. As a result, smooth communication can be achieved between the first user 120 and the second user 130.

[0160] Although not shown in Figure 18, suppose that the second user 130, while viewing the authorized person image 1101 included in the unauthorized person screen 720, moves the pointer of the second user 130 and points to a surface different from the surface pointed to by the pointer of the first user 120.

[0161] Under such a display situation, suppose that the second user 130 utters, for example, "I want to change the material of this surface." In this case, by visually checking the authority image 1101 included in the authority screen 640, the first user 120 can understand that the surface whose material the second user 130 wishes to change is the surface pointed to by the pointer of the second user 130. As a result, smooth communication can be achieved between the first user 120 and the second user 130.

[0162] 19 is a second diagram showing an example of a communication scene when the display system according to the second embodiment is used. As shown in FIG. 19 , in the case of the display system 1300, only the first user 120 has operation authority, and multiple users do not have operation authority at the same time.

[0163] Therefore, for example, even if the first user 120 and the second user 130 try to view the window side of the three-dimensional model in the virtual space, only the first user 120 can operate the three-dimensional model. In other words, in the case of the display system 1300 according to the second embodiment, it is possible to avoid a situation in which different operation instructions are given at the same time.

[0164] Furthermore, the first user 120 rotates the three-dimensional model 90 degrees counterclockwise by moving the controller 124 in a counterclockwise arc. This allows the first user 120 to view a desired side of the three-dimensional model in the virtual space by viewing the authorized person image 1201' included in the authorized person screen 640. Furthermore, the second user 130 can view a desired side of the three-dimensional model in the virtual space by viewing the authorized person image 1201' included in the non-authorized person screen 720.

[0165] <Summary> As is clear from the above explanation, the information processing device 110 according to the second embodiment: - sets operation authority for operating a three-dimensional model in a virtual space to the first user 120. - generates an authorized person image in which the three-dimensional model in the virtual space based on the viewpoint of the first user 120 is operated based on an operation instruction from the first user 120, and generates an authorized person screen including the authorized person image. - generates a non-authorized person screen including the authorized person image. - controls so that the authorized person screen is displayed on the first HMD 121 worn by the first user, and the non-authorized person screen is displayed on the information terminal 1301 viewed by the second user.

[0166] In this way, in the information processing device 110 according to the second embodiment, operation authority is set for a user wearing an HMD, and an authorized person image is shared among multiple users. As a result, according to the second embodiment, smooth communication can be realized when multiple users communicate while viewing the same three-dimensional model in a virtual space.

[0167] [Third Embodiment] In the first embodiment, the display system 100 has been described as having two combinations of an HMD, an audio output device, an audio input device, and a controller, but the number of combinations that the display system 100 has is not limited to two. For example, the display system 100 may have three or more combinations and be applied to communication scenes involving three or more people.

[0168] Similarly, in the second embodiment, the display system 1300 includes one combination of an HMD, an audio output device, an audio input device, and a controller, and one combination of an information terminal, an audio output device, an audio input device, and a controller. However, the number of combinations included in the display system 1300 is not limited to one. For example, the display system 1300 may include one combination of an HMD, an audio output device, an audio input device, and a controller, and multiple combinations of an information terminal, an audio output device, an audio input device, and a controller.

[0169] Alternatively, in the first embodiment described above, if the display system 100 has three or more combinations, one or more of the combinations may be replaced with a combination of an information terminal, an audio output device, an audio input device, and a controller.

[0170] [Other Embodiments] In the above embodiments, the virtual space image provided by the virtual space image providing unit 501 has been described as an image simulating a real space, but the virtual space image provided by the virtual space image providing unit 501 may be any image. For example, when multiple users perform molecular-level analysis of a material, a molecular-level image of the material may be provided.

[0171] In the first embodiment, a user who has been granted operation authority can switch between an operation instruction and a position specification instruction by operating a selector switch on a controller. However, the function of the information processing device 110 is not limited to this. For example, the information processing device 110 may be configured to change the display mode of a pointer in an authorized person image when the operation instruction is switched between an operation instruction and a position specification instruction.

[0172] In the first embodiment, the unauthorized user screen is described as displaying an authorized user image in the main area and an unauthorized user image in the sub-area. However, the unauthorized user screen may be configured to display an authorized user image in the sub-area and an unauthorized user image in the main area. Alternatively, the unauthorized user screen may be configured to switch between the authorized user image and the unauthorized user image based on a switching instruction from a user who does not have operating authority.

[0173] Furthermore, in each of the above embodiments, the HMD, the audio output device, and the audio input device are configured as separate entities, but the HMD, the audio output device, and the audio input device may also be configured as an integrated entity.

[0174] In addition, in each of the above embodiments, the audio output device, the audio input device, and the controller are configured to transmit and receive information to and from the information processing device via the HMD or the information terminal, but the audio output device, the audio input device, and the controller may be configured to transmit and receive information directly to and from the information processing device.

[0175] Furthermore, in the above second embodiment, the second user 130 is described as inputting a position designation instruction using the controller 134, but the second user 130 may also input a position designation instruction using, for example, an instruction device such as a mouse connected to the information terminal 1301.

[0176] In addition, in the above embodiments, the case where the display method is applied to virtual reality in which a three-dimensional model is displayed in a virtual space image has been described. However, the application of the display method is not limited to virtual reality, and the display method may be applied to augmented reality in which a virtual space image including a three-dimensional model is displayed in an area where the real space is visible.

[0177] The present disclosure is not limited to the configurations described in the above embodiments, but may be combined with other elements, etc. These aspects can be changed without departing from the spirit of the present disclosure, and can be appropriately determined depending on the application form.

[0178] This application claims priority based on Japanese Patent Application No. 2023-185779 filed on October 30, 2023, the entire contents of which are incorporated herein by reference.

[0179] 100: Display system 110: Information processing device 121: First HMD 122: Audio output device 123: Audio input device 124: Controller 131: Second HMD 132: Audio output device 133: Audio input device 134: Controller 435: Display device 445: Display device 501: Virtual space image providing unit 511: Authorized person image generating unit 512: Unauthorized person image generating unit 531: Authority setting unit 532: Control unit 540: Audio data transmitting / receiving unit 611, 612: Indicator 630: Authorized person image 640: Authorized person screen 710: Unauthorized person image 720: Unauthorized person screen 1300: Display system 1301: Information terminal 1511: Unauthorized person image generating unit 1521: Control unit

Claims

1. An information processing device having: an authority setting unit that sets operation authority for operating a three-dimensional model in a virtual space to any one of the users; an authority image generation unit that, when the operation authority is set to a first user, generates an authority screen including an authority image in which a three-dimensional model in the virtual space based on the viewpoint of the first user is operated based on an operation instruction from the first user; an unauthorized user image generation unit that generates an unauthorized user screen including an unauthorized user image showing a virtual space based on the viewpoint of a second user to whom the operation authority is not set, and the authorized user image; and a control unit that controls the authorized user screen to be displayed on a first HMD worn by the first user, and the unauthorized user screen to be displayed on a second HMD worn by the second user.

2. The information processing device of claim 1, wherein the authorized person image generation unit, when the operating authority is set to a second user, generates an authorized person image by operating a three-dimensional model in a virtual space based on the viewpoint of the second user based on operation instructions from the second user, and the non-authorized person image generation unit generates an non-authorized person image showing a virtual space based on the viewpoint of a first user to whom the operating authority is not set, and the control unit controls the authorized person screen to be displayed on a second HMD worn by the second user and the non-authorized person screen to be displayed on a first HMD worn by the first user.

3. The information processing device of claim 1 or 2, wherein the control unit controls the authorized person image generation unit to notify the viewpoint data received from the first HMD and the operation instructions by the first user when the operation authority is set to the first user, and the authorized person image generation unit generates the authorized person image by generating an image of a virtual space based on the viewpoint of the first user based on the viewpoint data received from the first HMD, and generating an image of a three-dimensional model in the virtual space based on the viewpoint of the first user based on the operation instructions by the first user.

4. The information processing device of claim 2, wherein the control unit controls the authorized person image generation unit to notify the viewpoint data received from the second HMD and the operation instructions by the second user when the operation authority is set to the second user, and the authorized person image generation unit generates the authorized person image by generating an image of a virtual space based on the viewpoint of the second user based on the viewpoint data received from the second HMD, and generating an image of a three-dimensional model in the virtual space based on the viewpoint of the second user based on the operation instructions by the second user.

5. The information processing device of claim 3, wherein the control unit controls the control unit to notify the authorized person image generation unit of a position specification instruction by the second user when the operating authority is set to the first user, and the authorized person image generation unit generates the authorized person image including an indicator indicating a position in a virtual space based on the viewpoint of the first user based on the position specification instruction by the second user.

6. The information processing device of claim 4, wherein the control unit controls the control unit to notify the authorized person image generation unit of a position designation instruction by the first user when the operating authority is set to a second user, and the authorized person image generation unit generates the authorized person image including an indicator indicating a position in a virtual space based on the viewpoint of the second user based on the position designation instruction by the first user.

7. An information processing device according to any one of claims 1 to 6, further comprising a transceiver unit that transmits the voice of the first user to the second HMD and transmits the voice of the second user to the first HMD.

8. An information processing device as described in any one of claims 1 to 7, wherein the unauthorized user image generating unit generates an unauthorized user screen including the authorized user image when the second user, to whom the operating authority has not been set, does not wear the second HMD, and the control unit controls the authorized user screen to be displayed on a first HMD worn by the first user and the unauthorized user screen to be displayed on a display device viewed by the second user.

9. The information processing device of claim 2, wherein the unauthorized user image generating unit generates an unauthorized user screen including the authorized user image when the first user, to whom the operating authority has not been set, does not wear the first HMD, and the control unit controls the authorized user screen to be displayed on a second HMD worn by the second user and the unauthorized user screen to be displayed on a display device viewed by the first user.

10. A display system comprising: the information processing device according to claim 1 or 2; and the first HMD and the second HMD according to claim 1 or 2.

11. A display system comprising: the information processing device according to claim 8; and the first HMD and the display device according to claim 8.

12. A display system comprising: the information processing device according to claim 9; and the second HMD and the display device according to claim 9.

13. A display method executed by a computer, comprising the steps of: setting operation authority for operating a three-dimensional model in a virtual space to any one of the users; when the operation authority is set to a first user, generating an authorized user screen including an authorized user image that operates a three-dimensional model in the virtual space based on the viewpoint of the first user based on an operation instruction from the first user; generating a non-authorized user screen including an unauthorized user image showing a virtual space based on the viewpoint of a second user to whom the operation authority is not set, and the authorized user image; and controlling the display of the authorized user screen on a first HMD worn by the first user, and the display of the non-authorized user screen on a second HMD worn by the second user.

14. A display program for causing a computer to execute the following steps: setting operation authority for operating a three-dimensional model in a virtual space to any one of the users; when the operation authority is set to a first user, generating an authorized user screen including an authorized user image that operates a three-dimensional model in the virtual space based on the viewpoint of the first user based on an operation instruction from the first user; generating a non-authorized user screen including a non-authorized user image showing a virtual space based on the viewpoint of a second user to whom the operation authority is not set, and the authorized user image; and controlling the display of the authorized user screen on a first HMD worn by the first user, and the non-authorized user screen on a second HMD worn by the second user.

Images