Information processing system, output method, and program

Abstract

To provide a user who is operating an avatar positioned in one virtual space with information related to another virtual space.SOLUTION: An information processing system transmits visual field information for an avatar positioned in a virtual space via a network to a terminal device. A virtual conference server 1 includes an avatar management unit that positions a plurality of avatars of a first user (instructor) in different virtual spaces; a virtual display control unit 26 that generates information on a virtual space that is not being displayed on a terminal device of the first user; and a communication control unit 23 that transmits the information on the virtual space to the terminal device of the first user.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present invention relates to an information processing system, an output method, and a program. [Background technology]

[0002] By communicating with a server via a personal computer (PC), users can participate in online meetings, take e-learning courses, etc. Participants can also display avatars placed in a virtual space on a virtual reality (VR) device, allowing them to hold meetings in the virtual space.

[0003] A technology has been devised in which participants wearing terminal devices can take lessons in a virtual space (see, for example, Patent Document 1). Patent Document 1 discloses a configuration in which one instructor operates multiple avatars to conduct multiple lessons in a virtual space. Summary of the Invention [Problem to be solved by the invention]

[0004] However, conventional technologies have a problem in that they lack a function for notifying a user who is operating an avatar placed in one virtual space of information about the other virtual space. For example, if a user who is displaying a virtual space from the viewpoint of an avatar placed in one virtual space wants to be called into the other virtual space, there is no function for notifying the user.

[0005] In view of the above-mentioned problems, the present invention provides a technique for notifying a user operating an avatar placed in one virtual space of information relating to the other virtual space. [Means for solving the problem]

[0006] In view of the above problems, the present invention provides an information processing system that transmits visibility information of avatars placed in a virtual space to a terminal device via a network, and includes an avatar management unit that places multiple avatars of a first user in different virtual spaces, a virtual display control unit that generates information about the virtual space that is not displayed on the terminal device of the first user, and a communication control unit that transmits information about the virtual space to the terminal device of the first user. [Effects of the Invention]

[0007] It is possible to provide a technique for notifying a user operating an avatar placed in one virtual space of information relating to the other virtual space. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram illustrating a configuration example of a virtual space system. [Figure 2A] FIG. 2 is a diagram illustrating a hardware configuration of an example of a virtual conference server. [Figure 2B] FIG. 1 is a diagram illustrating a hardware configuration of an example of a notebook PC. [Figure 3] FIG. 1 is a diagram illustrating a hardware configuration of an example of VR goggles. [Figure 4] FIG. 1 is a hardware configuration diagram of an example of a VR operation controller. [Figure 5] FIG. 1 is an example of a functional block diagram illustrating the functions of a virtual conference server, a notebook PC, VR goggles, and a VR operation controller in a virtual space system, divided into blocks. [Figure 6] FIG. 10 is a sequence diagram illustrating an example of a process performed by a lecturer to start a conference in a virtual conference space. [Figure 7] FIG. 10 is a sequence diagram illustrating an example of a process for a user to participate in a conference in a virtual conference space. [Figure 8] FIG. 10 is a diagram showing an example of a menu list screen displayed in the virtual conference space. [Figure 9]FIG. 10 is a diagram showing an example of notebook PC display data of a notebook PC displayed in a virtual conference space. [Figure 10] FIG. 1 is a diagram illustrating avatars arranged in three groups held in a virtual conference space. [Figure 11] 10 is a diagram illustrating an example of correspondence information between groups, participants, and instructor avatars. [Figure 12] 10 is an example of a diagram showing the field of view of instructor avatar Z1 in group 1. FIG. [Figure 13] FIG. 10 is a diagram showing an example of the field of view screen of a lecturer avatar on which a notification message indicating that the discussion is stalled is displayed. [Figure 14] FIG. 10 is a diagram illustrating the instructor pressing a notification message. [Figure 15] FIG. 10 is a diagram showing an example of a field of view screen on which a switch now button and a switch later button are displayed. [Figure 16] This is an example of a diagram showing the field of view screen of group 2 displayed by the VR goggles by pressing the switch button now. [Figure 17] This is an example of a diagram showing the field of view screen of group 1 displayed by the VR goggles later by pressing the switching button. [Figure 18] This is an example of a diagram showing the field of view screen of avatar E (Aomori-san) in group 2 when the instructor presses the switch button later. [Figure 19] 10 is an example of a sequence diagram illustrating a process in which an output control unit displays a notification message on the VR goggles of a lecturer. [Figure 20] 10 is an example of a sequence diagram illustrating a process when a lecturer responds to a notification message. [Figure 21] 10A and 10B are diagrams showing examples of statements made by avatars in a group; [Figure 22] FIG. 10 is a diagram showing an example of the field of view screen of an instructor avatar of Group 1 on which an inquiry notification message is displayed. [Figure 23] FIG. 10 is a diagram showing an example of a field of view screen on which a plurality of notification messages are simultaneously displayed. [Figure 24]10 is an example of a sequence diagram illustrating a process in which an output control unit notifies an instructor avatar of an inquiry from a group that is not currently displaying a view screen on the VR goggles and is not currently outputting audio data. [Figure 25] FIG. 10 is a diagram showing an example of an avatar switching button displayed on the instructor's VR goggles. [Figure 26] FIG. 10 is a diagram showing an example of a view screen displaying a list of instructor avatars. [Figure 27] 10 is an example of a sequence diagram illustrating a process of switching the view screen displayed by the instructor's VR goggles. [Figure 28] FIG. 10 is a diagram showing an example of a field of view screen including a notification button with a mark added. [Figure 29] 10 is an example of a sequence diagram illustrating a process in which the instructor causes the VR goggles to display a notification message again. [Figure 30] FIG. 10 is a diagram showing an example of the field of view screen of the instructor avatar of Group 1 on which the inquiry notification message is displayed again. DETAILED DESCRIPTION OF THE INVENTION

[0009] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS A virtual space system and an output method performed by the virtual space system will be described below with reference to the drawings as an example of an embodiment of the present invention.

[0010] <Terminology> View information is information that visualizes the state of the virtual space as seen from the viewpoint of an avatar. In this embodiment, view information is described using the term view screen. The VR goggles monitor the user's viewpoint using a gyro sensor and an acceleration sensor, and a 3D model constructed in the virtual space in the direction of the user's viewpoint is displayed on the VR goggles' display. Alternatively, the user can control the user's viewpoint using a VR operation controller. A 3D model is data that represents a three-dimensional surface using polygons or the like with three-dimensional coordinates. 3D models in this embodiment include desks, electronic whiteboards, and other users' avatars.

[0011] A virtual space is, for example, a 360° three-dimensional space reproduced by a computer. The term "virtual space" is used to describe a place where people interact and engage in social activities over a network, as if in the real world, and is likened to a physical space. In this embodiment, a conference room is primarily reproduced in the virtual space, but any three-dimensional space can be reproduced in the virtual space. In this embodiment, the virtual space may be a two-dimensional space as long as planar display data is displayed.

[0012] A pointer is a cursor that indicates the input position on a computer screen, and is, for example, an arrow-shaped icon. Pointers are usually arrow-shaped, but the shape of the cursor can change depending on the state of the computer or the position of the mouse pointer.

[0013] In this embodiment, the display device that enables the user to view the virtual space is referred to as a VR goggle. The VR goggle may be an AR (Augmented Reality) goggle or an MR (Mixed Reality) goggle.

[0014] Avatars are computer-generated representations of participants. By wearing VR goggles, participants can see the avatars of each participant in the virtual meeting space as if they were in real space.

[0015] <System configuration example> 1 is a system configuration diagram of a virtual space system 300. In the virtual space system 300, a virtual conference server 1 is communicably connected to Wi-Fi routers 3, 7, and 8 via a network N. Notebook PCs 4, 11, and 14 can also be communicably connected to the Wi-Fi routers 3, 7, and 8. The network N is, for example, the Internet, but may also be an on-premise network such as an in-house network.

[0016] The virtual conference server 1 is an information processing system having one or more information processing devices. The virtual conference server 1 stores data of a virtual conference space 8 in memory and executes a virtual conference app. Hereinafter, the service provided by the virtual conference app executed by the virtual conference server 1 will be referred to as the virtual conference service. The virtual conference space 8 is, for example, a 360° three-dimensional space that simulates a conference room in a virtual space. VR goggles 5, 12, and 15 (an example of a terminal device) display arbitrary display data 9 in the virtual space, and conference participants can view any viewpoint of the virtual conference space 8 using the VR goggles 5, 12, and 15. Participants can view the screens of devices placed in the conference room as if they were actually in the conference room.

[0017] The virtual conference space 8 has a screen acquired from the notebook PCs 4, 11, and 14 and synchronized with the screen of a notebook PC (hereafter referred to as a virtual notebook PC). The screen data of the notebook PCs 4, 11, and 14 is referred to as PC display data 9a and 9b. The virtual conference space 8 also manages virtual electronic whiteboard display data 9c and 9d as display data for a virtual electronic whiteboard. When there is no need to distinguish between the PC display data 9a and 9b and the virtual electronic whiteboard display data 9c and 9d, the data displayed in the virtual conference space 8 is referred to as display data 9.

[0018] The virtual conference space 8 has rooms, and conference participants (lecturer Z, users A and B) can enter each room using avatars. In Figure 1, there are rooms A and B. Display data 9 in rooms A and B can be shared by participants who enter the rooms. PC display data 9a and 9b may be controlled so that only the participant's own data is visible.

[0019] In Figure 1, instructor Z is using a laptop PC 4, VR goggles 5, and a VR operation controller 6. The laptop PC 4 and VR goggles 5 are connected to a Wi-Fi router 3 via wireless LAN so that they can communicate with each other. The VR goggles 5 and VR operation controller 6 are connected via Bluetooth (registered trademark, omitted below). The laptop PC 4 and VR goggles 5 are also connected via Bluetooth. A virtual conference app is running on the laptop PC 4.

[0020] User A uses a laptop PC 11, VR goggles 12, and a VR operation controller 13 in an environment where a Wi-Fi router 7 is installed. User B uses a laptop PC 14, VR goggles 15, and a VR operation controller 16 in an environment where a Wi-Fi router 8 is installed.

[0021] The laptop PCs 11 and 14 of users A and B and the VR goggles 12 and 15 are connected to Wi-Fi routers 7 and 8 via wireless LAN, and the VR goggles 12 and 15 and the VR operation controllers 13 and 16 are connected via Bluetooth. The laptop PCs 11 and 14 and the VR goggles 12 and 15 are also connected via Bluetooth. A virtual conference app is running on the laptop PCs 11 and 14 of users A and B.

[0022] <Hardware configuration example> <<Virtual conference server and laptop>> Fig. 2A is a diagram showing an example of the hardware configuration of the virtual conference server 1. As shown in Fig. 2A, the virtual conference server 1 is constructed by a computer 500, and includes a CPU 501, a ROM 502, a RAM 503, an HD (Hard Disk) 504, an HDD (Hard Disk Drive) controller 505, a display 506, an external device connection I / F (Interface) 508, a network I / F 509, a bus line 510, a keyboard 511, an optical drive 514, and a media I / F 516.

[0023] Of these, the CPU 501 controls the overall operation of the virtual conference server 1. The ROM 502 stores programs, such as IPL, used to drive the CPU 501. The RAM 503 is used as a work area for the CPU 501. The HD 504 stores various data, such as programs. The HDD controller 505 controls the reading and writing of various data from and to the HD 504 under the control of the CPU 501. The display 506 displays various information, such as a cursor, menu, window, text, or image. The external device connection I / F 508 is an interface for connecting various external devices. In this case, external devices include, for example, a USB (Universal Serial Bus) memory or a mouse. The network I / F 509 is an interface for data communication using a network. The bus line 510 is an address bus, a data bus, or the like, for electrically connecting the components, such as the CPU 501, shown in FIG. 2A.

[0024] The keyboard 511 is a type of input means having multiple keys used to input characters, numbers, various instructions, etc. The optical drive 514 controls reading and writing of various data from and to an optical storage medium 513, which is an example of a removable storage medium. The optical storage medium 513 may be a CD, a DVD, Blu-ray (registered trademark), etc. The media I / F 516 controls reading and writing (storing) of data from and to a storage medium 515, such as a flash memory. <<Notebook PC>> 2B is a diagram showing an example of the hardware configuration of the notebook PC 4, 11, 14. As shown in FIG. 2B, the notebook PC 4, 11, 14 is constructed by a computer 600, and includes a CPU 521, a ROM 522, a RAM 523, a HD (Hard Disk) 524, an HDD (Hard Disk Drive) controller 525, a display 526, an external device connection I / F (Interface) 528, a network I / F 529, a bus line 530, a keyboard 531, a pointing device 532, and a media I / F 536.

[0025] Of these, the CPU 521 controls the overall operation of the notebook PC 4, 11, or 14. The ROM 522 stores programs, such as an IPL, used to drive the CPU 521. The RAM 523 is used as a work area for the CPU 521. The HD 524 stores various data, such as programs. The HDD controller 525 controls the reading and writing of various data from and to the HD 524 under the control of the CPU 521. The display 526 displays various information, such as a cursor, menus, windows, characters, or images. The external device connection I / F 528 is an interface for connecting various external devices. In this case, external devices include, for example, a USB (Universal Serial Bus) memory or a printer. The network I / F 529 is an interface for data communication using a network. The bus line 530 is an address bus, a data bus, or the like, for electrically connecting the components, such as the CPU 521, shown in FIG. 2B.

[0026] The keyboard 531 is a type of input means having multiple keys used to input characters, numbers, various instructions, etc. The pointing device 532 is a type of input means for selecting and executing various instructions, selecting a processing target, moving a cursor, etc. The media I / F 536 controls reading and writing (storing) data from and to a recording medium 535 such as a flash memory.

[0027] <<VRゴーグル> > 3 is a hardware configuration diagram of the VR goggles 5, 12, and 15. The VR goggles 5, 12, and 15 each include a CPU 80, a main memory 81, a ROM 82, a display controller 86, a wireless LAN controller 88, an audio codec 90, and a video codec 93, all of which are connected via a bus 94.

[0028] The CPU 80 executes and processes the OS and control processing programs that are read from the ROM 82 to the main memory 81. The main memory 81 includes a DRAM, and is used as a work area for the CPU 80, etc.

[0029] The ROM 82 is previously written with an OS, a system program for when the power is turned on, and a program for controlling the VR goggles 5, 12, and 15.

[0030] A UART 83 is connected to the CPU 80. The UART 83 is an interface for transmitting and receiving serial data between the CPU 80 and the Bluetooth module 84, and is composed of a FIFO (First In, First Out), a shift register, and the like.

[0031] The Bluetooth module 84 is composed of an RF section and a baseband section, and is connected to an antenna 85, and performs wireless communication in accordance with the Bluetooth standard.

[0032] The display controller 86 performs D / A conversion of character data, graphic data, image data, etc., and also controls the display of this data on the LCD 87 .

[0033] The wireless LAN controller 88 executes a communication protocol that complies with the IEEE (Institute of Electrical and Electronics Engineers) 802.11ax standard, and controls communication with other devices by transmitting and receiving radio waves via the antenna 89.

[0034] An audio signal input from a microphone 91 is converted into audio data by an A / D (Analog / Digital) conversion circuit, and this audio data is coded by an audio codec 90 using the AAC (Advanced Audio Coding) method. Furthermore, AAC coded data received from an external device is decoded by the audio codec 90, converted to analog by a D / A conversion circuit, and the resulting audio is output from a speaker 92. A video codec 93 decodes compressed video data (for example, in accordance with ITU-T Recommendation H.264) received from an external device. Data is exchanged between the above components via a bus 94.

[0035] <<VR Operation Controller>> Figure 4 is a hardware configuration diagram of the VR operation controllers 6, 13, and 16. The VR operation controllers 6, 13, and 16 are connected to the CPU 110 and have a UART 117, a main memory 111, a ROM 112, a six-axis acceleration / angular velocity sensor 113, a menu display button 114, a pointer display button 115, and an OK button 116.

[0036] The CPU 110 executes and processes the control processing program read from the ROM 112 into the main memory 111. The main memory 111 has a DRAM and is used as a work area etc. of the CPU 110.

[0037] In the ROM 112, a system program at power-on, and programs for transmitting the press information of the menu display button 114, the pointer display button 115, and the OK button 116 via Bluetooth are pre-written.

[0038] The six-axis acceleration / angular velocity sensor 113 outputs measurement data of acceleration and angular velocity. The UART 117 is an interface for exchanging serial data between the CPU 110 and the Bluetooth module 118, and is composed of a FIFO, a shift register, etc. The Bluetooth module 118 is composed of an RF part and a baseband part, and an antenna 119 is connected thereto, and performs wireless communication conforming to the Bluetooth standard.

[0039] <Regarding Functions> Next, referring to Figure 5, the functional configuration of the virtual space system 300 will be described. Figure 5 is an example of a functional block diagram for separately explaining the functions of the virtual conference server 1, the notebook PCs 4, 11, 14, the VR goggles 5, 12, 15, and the VR operation controllers 6, 13, 16 in the virtual space system 300 in blocks.

[0040] <<Virtual Conference Server>> The virtual conference server 1 includes a virtual conference control unit 20, a whiteboard generation and management unit 21, a screen synchronization control unit 22, a communication control unit 23, a conference data storage unit 24, a room management unit 41, an output control unit 43, a display data generation unit 25, a virtual display control unit 26, a pointer position calculation unit 27, a user authentication unit 28, a user information storage unit 29, a voice recognition unit 42, and an avatar management unit 44. Each of these units included in the virtual conference server 1 is a function or a means for performing a function that is realized when any of the components shown in FIG. 2A operates in response to an instruction from the CPU 501 in accordance with a program loaded from the HD 504 to the RAM 503.

[0041] The virtual conference control unit 20 associates conference participants and devices connected to the conference (notebook PCs 4, 11, 14, VR goggles 5, 12, 15) with conference identification information (conference ID) and stores the information in the conference data storage unit 24. The virtual conference control unit 20 executes a remote conference with each device for each conference ID. The virtual conference control unit 20 also stores virtual electronic whiteboard display data 9c, 9d, PC display data 9a, 9b, audio data received from devices connected to the virtual conference, and the like in the conference data storage unit 24.

[0042] The whiteboard generating and managing unit 21 generates a virtual electronic whiteboard in the virtual conference space 8 and manages the virtual electronic whiteboard display data 9c and 9d.

[0043] When a participant uses a physical electronic whiteboard, the screen synchronization control unit 22 synchronizes the screen data of the electronic whiteboard with the virtual electronic whiteboard display data 9c, 9d so that they are the same. The screen synchronization control unit 22 operates as an application layer.

[0044] The communication control unit 23 is a means for transmitting information about the group to the instructor Z's VR goggles 5. The communication control unit 23 is also a means for transmitting the field of view information of the avatar generated by the virtual display control unit 26 to the instructor Z's VR goggles 5. Specifically, the communication control unit 23 transmits and receives data between the notebook PCs 4, 11, and 14 and the VR goggles 5, 12, and 15.

[0045] The conference data storage unit 24 stores the virtual electronic whiteboard display data 9c and 9d, the PC display data 9a and 9b, and audio data received from devices connected to the virtual conference. The conference data storage unit 24 also stores notification messages between groups.

[0046] The room management unit 41 manages conferences in the virtual conference space 8 in units called rooms. Multiple conferences can be held simultaneously in the virtual conference space 8. One conference is held in one room, and a virtual conference space 8 is constructed in each room.

[0047] The output control unit 43 outputs information about the virtual space whose field of view information is not displayed on the instructor's VR goggles 5 to the instructor's VR goggles 5. Specifically, the output control unit 43 analyzes user operations and causes the display data generation unit 25 to generate appropriate notification messages for the conference. For example, if there is no speech for a certain period of time in the conference, the output control unit 43 notifies instructor Z of the notification message. Furthermore, if instructor Z is busy, the output control unit 43 sends notification messages to the participants in the group on behalf of instructor Z.

[0048] The display data generation unit 25 generates the menus to be displayed in the virtual conference space 8, the pointer 170 instructed to be displayed by the VR operation controllers 6, 13, 16, and the like.

[0049] The virtual display control unit 26 is a means for displaying a pointer 170 indicating virtual electronic whiteboard display data in the virtual conference space 8 based on operation information transmitted from the VR operation controllers 6, 13, and 16. Specifically, the virtual display control unit 26 displays various display data in the virtual conference space 8. The virtual display control unit 26 displays, in the virtual conference space 8, the menu and pointer 170 generated by the display data generation unit 25, virtual electronic whiteboard display data 9c and 9d related to the virtual electronic whiteboard generated by the whiteboard generation and management unit 21, and PC display data 9a and 9b related to the notebook PCs 4, 11, and 14. The virtual display control unit 26 transmits the display data for the virtual conference space 8 to the VR goggles 5, 12, and 15 via the communication control unit 23, and the virtual conference space 8 is displayed on the VR goggles 5, 12, and 15.

[0050] The pointer position calculation unit 27 calculates the position (in the virtual conference space 8) of the pointer 170 pointed by the direction and movement of the VR operation controllers 6, 13, and 16.

[0051] The user authentication unit 28 compares the username and password received via the communication control unit 23 with the username and password stored in the user information storage unit 29, and authenticates the user participating in the virtual conference.

[0052] The user information storage unit 29 stores the usernames and passwords of users who will participate in the virtual conference.

[0053] The voice recognition unit 42 recognizes the utterances (voices) of users corresponding to each avatar in the virtual conference and records them together with the user's identification information. The voice data and the recognition results are associated and stored in the conference data storage unit 24. The voice recognition may be performed by a dedicated server.

[0054] The avatar management unit 44 is a means for arranging multiple instructor avatars Z1 to Z3 of a first user (an example of an instructor) in different virtual spaces. Specifically, the avatar management unit 44 generates avatars corresponding to users A and B, and an avatar corresponding to instructor Z. Multiple avatars are generated for instructor Z. The avatar management unit 44 manages the correspondence between groups and avatars.

[0055] <<Notebook PC>> The notebook PCs 4, 11, and 14 each have a virtual conference connection unit 30, a remote desktop control unit 31, a wireless LAN communication control unit 32, a Bluetooth communication control unit 33, a display control unit 34, and a data storage unit 35. Each of these units in the notebook PCs 4, 11, and 14 is a function or a means for performing the function, which is realized when any of the components shown in FIG. 2B operates in response to an instruction from the CPU 501 in accordance with a program (virtual conference application) loaded from the HD 504 to the RAM 503.

[0056] The virtual conference connection unit 30 connects to the virtual conference service of the virtual conference server 1 and transmits a username and password, which are authentication data. The virtual conference connection unit 30 also manages the URL and conference ID of the virtual conference space 8 received from the virtual conference service, and connects to the URL of the virtual conference space 8 to participate in the virtual conference. Furthermore, the virtual conference connection unit 30 transmits the URL and conference ID of the virtual conference space 8 to the VR goggles 5, 12, and 15 via the Bluetooth communication control unit 33.

[0057] The remote desktop control unit 31 transmits the screen data of the notebook PCs 4, 11, and 14 to the virtual conference service to display the data in the virtual conference space 8.

[0058] The wireless LAN communication control unit 32 executes communication with the virtual conference server 1 via the Wi-Fi routers 3, 7, and 8.

[0059] The Bluetooth communication control unit 33 executes communication conforming to the Bluetooth standard with the VR goggles 5, 12, and 15. The display control unit 34 performs control for displaying screen data on a display (LCD).

[0060] The data storage unit 35 stores the address (URL) of the virtual conference service provided by the virtual conference server 1, the URL of the virtual conference space 8 received from the virtual conference service, the conference ID, the username and password of the user using the notebook PC 4, 11, 14, etc.

[0061] <<VRゴーグル> > The VR goggles 5, 12, and 15 each have a display control unit 100, a wireless LAN communication control unit 101, a Bluetooth communication control unit 102, a virtual conference connection unit 103, a button information transfer unit 104, and a data storage unit 105. Each of these units in the VR goggles 5, 12, and 15 is a function or a means for performing the function that is realized when any of the components shown in Fig. 3 operates in response to an instruction from the CPU 80 in accordance with a program (virtual conference application) loaded from the ROM 82 to the main memory 81.

[0062] The display control unit 100 controls the display of display data transmitted from the virtual conference service on the LCD 87. The wireless LAN communication control unit 101 executes communication with the virtual conference server 1 via Wi-Fi routers 3, 7, and 8.

[0063] The Bluetooth communication control unit 102 executes communication conforming to the Bluetooth standard with the VR operation controllers 6, 13, and 16. The virtual conference connection unit 103 uses the URL and conference ID of the virtual conference service transmitted from the laptop PC 4 to perform control for connecting to the virtual conference service executed by the virtual conference server 1.

[0064] The button information transfer unit 104 receives button press information from the VR operation controllers 6, 13, and 16, and transmits information regarding the pressed button to the virtual conference server 1 via the wireless LAN communication control unit 101.

[0065] The data storage unit 105 stores the address (URL) of the virtual conference service provided by the virtual conference server 1, the conference ID, the user name of the user using the VR goggles 5, 12, 15, and the password, etc.

[0066] <<VR operation controller>> The VR operation controllers 6, 13, and 16 include a Bluetooth communication control unit 120, a button information transmission unit 121, and an orientation / movement detection unit 122. These functions of the VR operation controllers 6, 13, and 16 are functions or means realized by operating according to instructions from the CPU 110 in which a program developed from the ROM 112 shown in FIG. 4 is expanded to the main memory 111.

[0067] The Bluetooth communication control unit 120 performs communication compliant with the Bluetooth standard with the VR goggles 5, 12, 15. The button information transmission unit 121 passes the press information of the menu display button 114, the pointer display button 115, and the confirmation button 116 to the Bluetooth communication control unit 120.

[0068] The orientation / movement detection unit 122 detects the orientation (posture in three-dimensional space) and movement of the VR operation controllers 6, 13, and 16 from the acceleration and angular velocity data obtained from the six-axis acceleration / angular velocity sensor 113, and passes this information to the Bluetooth communication control unit 120.

[0069] <Operation or process> Next, with reference to Fig. 6, a procedure will be described in which instructor Z and user A use laptop PCs 4, 11 and VR goggles 5, 12 to participate in the virtual conference space 8 in the virtual conference server 1 and start a conference in the virtual space. Fig. 6 is a sequence diagram illustrating the process until instructor Z starts a conference in the virtual conference space 8.

[0070] When lecturer Z, the conference organizer, starts the virtual conference app on his laptop PC 4, the virtual conference connection unit 30 connects to the virtual conference service on the virtual conference server 1 and displays a screen for entering a username and password. In the following, no distinction is made between the virtual conference server 1 and the virtual conference service, and the explanation will be centered around the virtual conference service. The URL of the virtual conference service is pre-registered in the laptop PC 4 as setting data for the virtual conference app.

[0071] When instructor Z enters his / her username and password, the virtual conference connection unit 30 transmits the username and password to the virtual conference service via the wireless LAN communication control unit 32. The virtual conference service authenticates the user and transmits an authentication OK response to the laptop PC 4.

[0072] S1: When the virtual conference connection unit 30 receives an authentication OK response via the wireless LAN communication control unit 32, the display control unit 34 of the virtual conference app running on the laptop PC 4 displays a start button and a join button for the virtual conference. Since instructor Z is the organizer, he or she selects the start button.

[0073] S2: The virtual conference connection unit 30 transmits a command to start the virtual conference to the virtual conference service via the wireless LAN communication control unit 32. This transmission uses, for example, HTTP.

[0074] S3: When the virtual conference control unit 20 of the virtual conference service receives this start command via the communication control unit 23, it generates a virtual conference space 8 and a conference ID.

[0075] S4: The communication control unit 23 transmits the URL of the virtual conference space 8 and the conference ID to the notebook PC 4.

[0076] S5: When the wireless LAN communication control unit 32 of the notebook PC 4 receives this data, the virtual conference connection unit 30 connects to the URL of the virtual conference space 8 via the wireless LAN communication control unit 32 and sends a conference participation command including the received conference ID.

[0077] S6: The virtual conference control unit 20 of the virtual conference service compares this conference ID with the conference ID generated in step S3 and, if it confirms a match, allows the notebook PC 4 to connect to the virtual conference space 8 (participation in the virtual conference corresponding to the conference ID).

[0078] S7: The virtual conference control unit 20 transmits a response OK to the notebook PC 4 via the communication control unit 23.

[0079] S8: When the wireless LAN communication control unit 32 of the notebook PC 4 receives the response OK, the virtual conference connection unit 30 sends the URL of the virtual conference space 8 and the conference ID to instructor Z's VR goggles 5 via Bluetooth via the Bluetooth communication control unit 33.

[0080] S9: When the Bluetooth communication control unit 102 of the VR goggles 5 receives this data, the virtual conference connection unit 103 connects to the URL of the virtual conference space 8 via the wireless LAN communication control unit 101 and sends a conference participation command including the received conference ID.

[0081] S10: The virtual conference control unit 20 of the virtual conference service compares this conference ID with the conference ID generated in step S3 and, if it confirms a match, allows connection between the VR goggles 5 and the virtual conference space 8 (participation in the virtual conference corresponding to the conference ID).

[0082] S11: The virtual conference control unit 20 transmits a response OK to the VR goggles 5 via the communication control unit 23.

[0083] S12: The virtual display control unit 26 of the virtual conference service converts the 3D model of the virtual conference space 8 into a view screen seen by the instructor's avatar, and transmits it to the VR goggles 5 via the communication control unit 23. The view screen may include display data 9.

[0084] S13: When the wireless LAN communication control unit 101 of the VR goggles 5 receives the screen information of the field of view screen, the display control unit 100 displays the received field of view screen of the virtual conference space 8 on the LCD 87.

[0085] S14: Next, the display control unit 34 of the virtual conference application of the notebook PC 4 displays a selection button for the remote desktop.

[0086] S15: Instructor Z presses the selection button on the remote desktop.

[0087] S16: The remote desktop control unit 31 transmits the screen data of the notebook PC 4 to the virtual conference service via the wireless LAN communication control unit 32.

[0088] S17: The virtual display control unit 26 of the virtual conference service displays the screen data received by the communication control unit 23 in the virtual conference space 8 (PC display data 9a).

[0089] S18: The virtual display control unit 26 of the virtual conference service transmits the PC display data 9a to the VR goggles 5.

[0090] S19: The display control unit 100 of the VR goggles 5 causes the LCD 87 to display this PC display data 9a.

[0091] With the above steps, instructor Z has joined the virtual conference, and the view screen of the virtual conference space 8 is displayed on the VR goggles 5 (the screen data of the laptop PC 4 is also displayed). Note that the virtual display control unit 26 of the virtual conference service controls the screen data of the laptop PC 4 displayed in the virtual conference space 8 so that it is displayed only on instructor Z's VR goggles 5.

[0092] <<Users A and B participate in a virtual meeting>> Next, instructor Z sends the URL of the virtual conference space 8 and the conference ID from his / her laptop PC 4 to the laptop PCs of the other conference participants by email or instant message. When user A, a conference participant, starts the virtual conference app on his / her laptop PC 11, the virtual conference connection unit 30 connects to the virtual conference service and displays a screen for entering a username and password. The URL of the virtual conference service is pre-registered in laptop PC 11 as setting data for the virtual conference app.

[0093] When User A enters his / her username and password, the virtual conference connection unit 30 sends this data to the virtual conference service. The virtual conference service authenticates the user and sends an authentication OK response to the laptop PC 11. The virtual conference app on the laptop PC 11 displays a button to start the virtual conference and a button to join.

[0094] FIG. 7 is a sequence diagram illustrating the process up to when user A participates in a conference in the virtual conference space 8. In FIG.

[0095] S21: User A is a participant, so he selects the participation button.

[0096] S22: The display control unit 34 of the notebook PC 11 displays a screen for inputting the URL of the virtual conference space 8 and the conference ID.

[0097] S23: User A enters the URL and conference ID of the virtual conference space 8 notified by instructor Z and presses the OK button.

[0098] S24: The virtual conference connection unit 30 transmits a connection command including this conference ID to the URL of the virtual conference space 8 via the wireless LAN communication control unit 32.

[0099] S25: The virtual conference control unit 20 of the virtual conference service collates this conference ID with the conference ID generated in step S3, and if they match, permits connection between the notebook PC 11 and the virtual conference space 8.

[0100] S26: The virtual conference control unit 20 transmits a response OK to the notebook PC 11 via the communication control unit 23.

[0101] S27: Next, the Bluetooth communication control unit 33 of the notebook PC 11 transmits the URL of the virtual conference space 8 and the conference ID to the VR goggles 12 of the user A via Bluetooth.

[0102] S28: When the Bluetooth communication control unit 102 of the VR goggles 12 receives this data, the virtual conference connection unit 103 sends a connection command including this conference ID to the URL of the virtual conference space 8 via the wireless LAN communication control unit 101.

[0103] S29: The virtual conference control unit 20 of the virtual conference service collates this conference ID with the conference ID generated in step S3, and if they match, permits connection between the VR goggles 12 and the virtual conference space 8.

[0104] S30: The virtual conference control unit 20 transmits a response OK to the VR goggles 12 via the communication control unit 23.

[0105] S30-2: The virtual display control unit 26 of the virtual conference service converts the 3D model of the virtual conference space 8 into a view screen seen by the avatar of user A, and transmits it to the VR goggles 12 via the communication control unit 23. The view screen may include display data 9.

[0106] When the wireless LAN communication control unit 101 of the VR goggles 12 receives the screen information of the field of view screen, the display control unit 100 displays the field of view screen of the virtual conference space 8 received on the LCD 87.

[0107] When the virtual conference connection unit 30 of the virtual conference application operating on the notebook PC 11 connects to the virtual conference space 8, the display control unit 34 displays the selection button for the remote desktop.

[0108] User A selects the selection button for the remote desktop.

[0109] The remote desktop control unit 31 of the notebook PC 11 transmits the screen data of the notebook PC 11 to the virtual conference service via the wireless LAN communication control unit 32.

[0110] The virtual display control unit 26 of the virtual conference service displays the screen data of the notebook PC 11 in the virtual conference space 8 (PC display data 9b).

[0111] The virtual display control unit 26 of the virtual conference service transmits the PC display data 9b to the VR goggles 12.

[0112] The display control unit 100 of the VR goggles 12 causes the PC display data 9b to be displayed on the LCD 87.

[0113] Note that the virtual display control unit 26 of the virtual conference service controls so that the screen data of the notebook PC 11 displayed in the virtual conference space 8 is displayed only on the VR goggles 12 of user A.

[0114] Other participants in the conference also perform the same operations as user A above, and connect the notebook PCs and VR goggles of each participant to the virtual conference space 8.

[0115] <Operation of VR operation controller> When the instructor Z or the users A and B press the menu display button 114 on the VR operation controller 6, 13, 16, the Bluetooth communication control unit 120 of the VR operation controller 6, 13, 16 transmits a menu display command to the VR goggles 5, 12, 15 via Bluetooth.

[0116] The wireless LAN communication control unit 101 of the VR goggles 5, 12, and 15 sends this command to the virtual conference service. When the virtual conference service receives this command, the display data generation unit 25 generates a menu list related to the virtual conference and displays it in the virtual conference space 8. An example of this menu list is shown in Figure 8.

[0117] 8 shows a menu list screen 200 displayed in the virtual conference space 8. The menu list screen 200 has a participant list button 201, a whiteboard generation button 202, and a settings button 203. The participant list button 201 is a button for displaying a list of participants currently attending a conference identified by a conference ID. The whiteboard generation button 202 is a button for generating a virtual electronic whiteboard in the virtual conference space 8. The settings button 203 is a button for displaying various setting screens such as language and character size.

[0118] Furthermore, when instructor Z or users A and B press the pointer display button 115 on the VR operation controller 6, 13, or 16, the Bluetooth communication control unit 120 of the VR operation controller 6, 13, or 16 transmits a pointer display command, including information about the orientation and movement of the VR device, to the VR goggles 5, 12, or 15 via Bluetooth. The wireless LAN communication control unit 101 of the VR goggles 5, 12, or 15 transmits this command to the virtual conference service. Upon receiving this command, the pointer position calculation unit 27 calculates the display position of the pointer 170 based on the information about the orientation and movement of the VR operation controller 6, 13, or 16. The virtual display control unit 26 displays the pointer 170 at the calculated display position in the virtual conference space 8.

[0119] While the user is pressing the pointer display button 115, the Bluetooth communication control unit 120 of the VR operation controller 6, 13, 16 periodically (for example, every 100 milliseconds) transmits a pointer display command including information about the orientation and movement of the device to the VR goggles 5, 12, 15. The wireless LAN communication control unit 101 of the VR goggles 5, 12, 15 transmits this command to the virtual conference service. The pointer position calculation unit 27 calculates the display position of the pointer 170 based on the information about the orientation and movement of the VR operation controller 6, 13, 16. When the pointer 170 moves and the display of the virtual conference space 8 changes, the virtual display control unit 26 of the virtual conference service transmits display data of the virtual conference space 8 to the VR goggles 5, 12, 15. Therefore, the pointer 170 is controlled to move within the virtual conference space 8.

[0120] When instructor Z or users A and B press the confirm button 116 on the VR operation controller 6, 13, 16, the Bluetooth communication control unit 120 of the VR operation controller 6, 13, 16 sends a confirm command via Bluetooth to the VR goggles 5, 12, 15. The wireless LAN communication control unit 101 of the VR goggles 5, 12, 15 sends this command to the virtual conference service. When the virtual conference service receives this command, it determines that the menu item corresponding to the display position of the pointer 170 has been selected (pressed), and performs an operation according to that menu.

[0121] When instructor Z (or user A or B) moves the pointer 170 to the whiteboard generation button 202 shown in FIG. 8 and presses the confirm button 116, the VR operation controllers 6, 13, and 16 send a confirm command to the virtual conference service via the VR goggles 5, 12, and 15. When the virtual conference service receives this command, the whiteboard generation and management unit 21 generates a virtual electronic whiteboard, and the virtual display control unit 26 displays it in the virtual conference space 8. Note that the screen when the virtual electronic whiteboard is generated is blank. In other words, blank virtual electronic whiteboard display data is displayed in the virtual conference space 8.

[0122] <Generating virtual electronic whiteboard display data> On this virtual electronic whiteboard, users can display screen data from laptop PCs 4, 11, and 14, as well as any display content (photos, graphics, etc.). In FIG. 1, the screens of two virtual electronic whiteboards (virtual electronic whiteboard display data 9c and virtual electronic whiteboard display data 9d) are displayed. The virtual electronic whiteboard display data 9c displays screen data from laptop PC 14 operated by User B, who is participating in the virtual conference. The virtual electronic whiteboard display data 9d displays display data that is not screen data from the laptop PCs 4, 11, and 14 (keyboard input, voice recognition results, handwritten data, etc.).

[0123] The following describes the process and operation of displaying the screen data of user B's laptop PC 14 on the virtual electronic whiteboard. When user B starts the virtual conference app on the laptop PC 14 and performs the same operation as user A described above, the laptop PC 14 and VR goggles 15 connect to the virtual conference space 8. When user B then selects the remote desktop selection button displayed on the laptop PC 14, the screen data of the laptop PC 14 is sent to the virtual conference service, and the screen data of the laptop PC 14 is displayed in the virtual conference space 8.

[0124] 9 shows the laptop PC display data 210 of the laptop PC 14 displayed in the virtual conference space 8. In the laptop PC display data 210, a movement bar 211 is added to a screen display area 212 in which the screen data of the laptop PC 14 in the virtual conference space 8 is displayed. The virtual electronic whiteboard display data 9c is displayed in the screen display area 212. When user B presses the pointer display button 115 on the VR operation controller 16 and holds the pointer 170 over the movement bar 211, the screen display area 212 of the laptop PC 14 becomes movable. When user B moves the pointer 170 while pressing the pointer display button 115, the screen display area 212 of the laptop PC 14 also moves accordingly.

[0125] When user B performs this operation to overlay screen display area 212 of notebook PC 14 on the virtual electronic whiteboard (blank virtual electronic whiteboard display data), virtual display control unit 26 of the virtual conference service copies the screen data of notebook PC 14 onto the screen of the virtual electronic whiteboard, and the screen data of notebook PC 14 (virtual electronic whiteboard display data 9c) is displayed in virtual conference space 8. In addition, virtual display control unit 26 deletes screen display area 212 and movement bar 211 on which the screen data of notebook PC 14 was displayed.

[0126] <Group work in a virtual meeting space> Next, group work in the virtual conference space 8 will be described with reference to FIG. 10. Group work is work in which multiple people form a group, discuss and create work based on a theme, and finally present their conclusions or deliverables. Group work can also be simply called a meeting. FIG. 10 schematically shows avatars arranged in three groups 1 to 3 held in the virtual conference space 8.

[0127] Participants are divided into three groups and conduct group work. The virtual space for each group corresponds to virtual conference space 8. Avatars A to K are avatars of participants (an example of a second user). Hereinafter, users will be referred to as participants in the group work. Avatars A to D are assigned to group 1 (an example of a first virtual space), avatars E to H to group 2 (an example of a second virtual space), and avatars I to K to group 3. Each group may have one or more avatars. The assignment is performed, for example, when participants log in to the virtual conference service. There is one instructor Z (an example of a first user). The same instructor Z chairs the group work for groups 1 to 3. Instructor Z also participates as an avatar, but to distinguish between them, they are referred to as instructor avatars Z1 to Z3.

[0128] For ease of explanation, the VR goggles used by the users of group 2 will be referred to as VR goggles 12, and the VR goggles used by the users of group 3 will be referred to as VR goggles 15.

[0129] Each avatar A to K is created when a participant first uses the virtual conference service, customizing the participant's appearance by selecting from several templates, such as face and clothing. While participating in the conference, the participant's name is displayed above each avatar A to K.

[0130] In the real world, instructor Z can only participate in one group at a time, but in virtual conference space 8, it is possible to generate multiple avatars of himself, so we will explain an example scene in which instructor avatars Z1 to Z3 exist in each group at the same time.

[0131] As shown in FIG. 11, the avatar management unit 44 manages groups, participants, and instructor avatars in association with each other. FIG. 11 shows association information between groups, participants, and instructor avatars. First, participants 1 to 11 log in to the groups assigned to them, so participants 1 to 11 are associated with the groups. Instructor Z places his / her instructor avatars 1 to 3 in each group. Instructor avatars 1 to 3 correspond to instructor avatars Z1 to Z3 in FIG. 10. The avatar management unit 44 places avatars A to K of participants 1 to 11 in virtual spaces corresponding to each group (virtual conference spaces 8 prepared for each group), and places instructor avatars Z1 to Z3 in the corresponding virtual spaces.

[0132] The virtual display control unit 26 controls the avatars in the group, the instructor avatar, the desk, the electronic blackboard ( A view screen is generated by perspectively projecting 3D models such as virtual electronic whiteboard display data (9c, 9d) and a notebook PC (PC display data 9a, 9b). The view screen is displayed on the VR goggles 5, 12, 15 of each participant and instructor Z. Regarding audio data, the communication control unit 23 regards the participants and instructor Z associated with a group as one session and transmits the audio of a participant in the group to all participants in the same group and instructor Z. The communication control unit 23 transmits instructor Z's audio data to the VR goggles of all participants in the group. Instructor Z may also be able to specify any participant to send audio data (private conversation).

[0133] Only one group's audio data and view screen are sent to Instructor Z's VR goggles 5 at a time. As will be described later, Instructor Z can switch between these groups at will. Of multiple groups (multiple virtual spaces), the group whose view screen and audio data are being sent to the instructor's VR goggles 5 may be referred to as the group that the instructor is "moderating." Moderating may also be referred to as the instructor "operating" the VR goggles 5. The instructor avatar of the group whose view screen is currently being displayed and whose audio data is being output on the VR goggles 5 is referred to as the "master." The instructor always operates the master.

[0134] In addition, instructor Z may be able to set the audio data and view screens of multiple groups to be transmitted simultaneously to his / her VR goggles 5. Instructor Z may be able to set the audio data and view screens of different groups to be transmitted to his / her VR goggles 5.

[0135] The instructor avatars Z1 to Z3 may or may not be displayed to participants in a group to which the audio data and view screen are not transmitted to the VR goggles 5 of instructor Z. By not displaying them, participants can know that instructor Z is currently moderating another group. Alternatively, even if instructor avatars Z1 to Z3 are displayed, they may be displayed in a manner that makes it clear that instructor Z is currently moderating another group.

[0136] FIG. 12 shows the field of view of instructor avatar Z1 in group 1. The field of view of instructor avatar Z1 is the field of view screen displayed on instructor Z's VR goggles 5. More specifically, the field of view screen is an image projected in the direction instructor Z is looking in the virtual 360° virtual space in which group 1 is constructed. Therefore, the field of view screen can display any space within 360°. Avatars A to D appear in the field of view of the instructor avatar. The explanation will be given assuming that instructor Z is initially operating instructor avatar Z1 in group 1. That is, the field of view screen of group 1 is displayed on instructor Z's VR goggles 5, and audio data of group 1 is output.

[0137] The view screen displays avatars A to D doing group work, as well as an avatar switching button 131 and a notification button 132. The avatar switching button 131 is a button for switching the view screen displayed on instructor Z's VR goggles 5 and the audio data output to those of another group. The notification button 132 is a button that displays that there is a notification for instructor avatar Z1 of the group whose VR goggles 5 is currently displaying the view screen and outputting audio data. The view screen that can be seen by other instructor avatars Z2 and Z3 may be the same as that shown in FIG. 12 (however, the avatars in view and the state of the notification button will differ).

[0138] <Notification to the instructor avatar when no conversation is taking place in Group 2> For example, even if there is a group where no conversation is taking place, if Instructor Z is not the moderator of that group, Instructor Z will not be able to notice. In such cases, it would be convenient if a notification to that effect was displayed on the field of view screen.

[0139] One method for detecting that no conversation has taken place for a certain period of time is for the output control unit 43 to use the voice recognition results of the voice recognition unit 42. When an avatar speaks within a group, the voice recognition unit 42 recognizes the voice data and can therefore determine that a speech has taken place within the group, and when the voice recognition unit 42 is unable to recognize the voices of the participants within the group for a certain period of time, it can be determined that no speech has taken place within the group during that time.

[0140] When the output control unit 43 detects that no conversation has taken place within a group for a certain period of time, it requests the virtual display control unit 26 of the virtual conference server to add a notification message to the field of view screen of instructor Z's VR goggles 5, indicating that the discussion is stalling. Upon receiving the request to add the notification message, the virtual display control unit 26 displays the notification message on the field of view screen of the group chaired by instructor Z. In this way, instructor Z can confirm from the notification message on his or her field of view screen that the discussion is stalling (no comments) in other groups that are currently being displayed on the field of view screen but are not outputting audio data.

[0141] FIG. 13 is an example of the field of view screen of instructor avatar Z1 on which a notification message 133 indicating that the discussion is stalled is displayed. The notification message 133, "The discussion is stalled. Group 2," is displayed at the top of the field of view screen. Instructor Z confirms from the notification message 133 that the discussion in Group 2 is stalled and presses the notification message 133. The notification message 133, "The discussion is stalled. Group 2," may be output by voice (the notification message 133 may or may not be displayed).

[0142] 14 is a diagram illustrating instructor Z pressing the notification message 133. Instructor Z operates the VR operation controller 6 to move the pointer 170 to the notification message 133 and presses the confirm button 116. When the notification message 133 is pressed, the virtual display control unit 26 displays a switch now button 134 and a switch later button 135.

[0143] 15 is an example of a field of view screen on which a switch now button 134 and a switch later button 135 are displayed. The switch now button 134 and the switch later button 135 are displayed on the field of view screen for group 1 displayed by instructor Z's VR goggles 5, so instructor Z can display the field of view screen for group 2 by pressing the switch now button 134. FIG. 16 shows the case where the switch now button 134 is pressed, and FIG. 17 shows the case where the switch later button 135 is pressed.

[0144] FIG. 16 shows the field of view screen for Group 2 displayed by the VR goggles 5 upon pressing the Switch Now button 134. Because instructor Z pressed the Switch Now button 134 displayed in response to the notification message regarding Group 2, the virtual display control unit 26 displays the field of view screen for Group 2 on the VR goggles 5. In a setting in which the instructor avatar Z2 is not displayed in groups that instructor Z is not moderating, the virtual display control unit 26 places the instructor avatar Z2 in Group 2. The virtual display control unit 26 displays a 3D model of Group 2 on the VR goggles 5. Therefore, this field of view screen is the field of view screen seen by instructor avatar Z2 of Group 2. Therefore, avatars E to H are displayed. Note that if no conversation has taken place in Group 3 for a certain period of time, a list of Groups 2 and 3 may be displayed by pressing the Switch Now button 134, and instructor Z may select Group 2 or 3 to display from the list.

[0145] FIG. 17 shows the field of view screen of Group 1 displayed by the VR goggles 5 when the later switch button 135 is pressed. Since Instructor Z is still moderating Group 1, this field of view screen displays a 3D model of Group 1 on the VR goggles 5. Because the later switch button 135 was pressed (an example of postponing a response), the virtual display control unit 26 erases the notification message 133 and displays a mark 136 indicating that a notification is available on the notification button 132. When the later switch button 135 is pressed, the output control unit 43 stores the notification message in the conference data storage unit 24. The conference data storage unit 24 stores the notification message itself, the name of the group that did not speak for a certain period of time, the time, etc.

[0146] In addition, when a certain time has passed since the notification message was displayed as shown in FIG. 14, the virtual display control unit 26 hides the notification message 133 and switches to displaying a mark 136 indicating that a notification is available on the notification button 132, even if the instructor Z does not press the switch button 135 later.

[0147] FIG. 18 shows the field of view screen of avatar E (Aomori-san) of group 2 when instructor Z presses the "later" switch button 135. As an example, FIG. 18 shows the field of view screen of avatar E, Aomori-san, of group 2. Because instructor Z pressed the "later" switch button 135, the output control unit 43 outputs audio data so that it can be heard from the direction of instructor avatar Z2. The virtual display control unit 26 may move the mouth of instructor avatar Z2 or display a statement from instructor avatar Z2 in a speech bubble or the like. The content of the statement is, for example, "I will join the discussion later." "I will join the discussion later" is a standard phrase generated by the output control unit. Because instructor Z's voice will be transmitted to group 1, making it a standard phrase prevents instructor Z from making any statements. A selection of standard phrases may be displayed on the field of view screen of group 1, from which instructor Z may select.

[0148] As a result of the above, avatars E to H in group 2 who have not spoken for a certain period of time can share with the other avatars that the instructor will join the discussion later.

[0149] In FIG. 18, instructor avatar Z2 is displayed in group 2. In this way, when the switching button 135 is pressed later, the virtual display control unit 26 may temporarily display instructor avatar Z2. Alternatively, the virtual display control unit 26 may display an avatar for the output control unit (an avatar for a bot) and output a statement from instructor Z as audio data in the form of a message. Alternatively, the virtual display control unit 26 may output only the generated audio without displaying instructor avatar Z2 or an avatar for a bot.

[0150] 14 to 18, the output control unit 43 detects that speech is stalled within the group, but the output control unit 43 may detect that speech is stalled for each participant and notify the instructor avatar Z1.

[0151] <Action or Processing> 19 is a sequence diagram illustrating the process in which the output control unit 43 displays a notification message on the VR goggles 5 of instructor Z. As a premise for the explanation of FIG. 19, instructor avatar Z1 of instructor Z is placed in group 1, and avatars of one or more participants are placed in group 2. If no utterances of a participant are detected in group 2, the virtual display control unit 26 generates a message indicating that no utterances of a participant are detected in group 2, and the communication control unit 23 transmits the message to the instructor's VR goggles 5 displaying the field of view information of group 1.

[0152] S51: The output control unit 43 always waits for conversation between avatars for each group. The output control unit 43 has a timer for each group, and resets the timer when there is conversation. When the timer times out, it determines that a certain period of time has passed without conversation.

[0153] S52: If there is no conversation for a certain period of time, a request to generate a notification message to the effect that the discussion is stalled is sent to the display data generating unit 25.

[0154] S53: Upon receiving the request, the display data generating unit 25 generates a notification message to the effect that the discussion is stalled.

[0155] S54: The display data generation unit 25 requests the virtual display control unit 26 to generate a field of view screen on which the notification message is combined.

[0156] S55: The virtual display control unit 26 generates a view screen on which the notification message is combined.

[0157] S56: The virtual display control unit 26 transmits screen information of the field of view screen onto which the notification message is combined to the VR goggles 5 via the communication control unit 23. The wireless LAN communication control unit 32 of the VR goggles 5 receives the screen information of the field of view screen, and the display control unit 34 displays it on the LCD 87 of the VR goggles 5 (see FIG. 13).

[0158] FIG. 20 is a sequence diagram illustrating the process when instructor Z responds to the notification message. S61: Instructor Z points the pointer 170 to the notification message 133 and presses the confirm button 116 (see FIG. 14).

[0159] S62-1: The VR operation controller 6 receives a press of the confirmation button 116, and the button information transmission unit 121 transmits a notification message to the VR goggles 5 via the Bluetooth communication control unit 120 to the effect that the confirmation button 116 has been pressed. S62-2: The wireless LAN communication control unit 101 of the VR goggles 5 transmits to the virtual conference server 1 a notification that the notification message has been pressed.

[0160] S63: The communication control unit 23 of the virtual conference server receives the notification message that the button has been pressed, and the virtual conference control unit 20 transmits a request to generate the now switch button 134 and the later switch button 135 to the display data generation unit 25.

[0161] S64: In response to this request, the display data generating unit 25 generates the now switch button 134 and the later switch button 135.

[0162] S65: Furthermore, the display data generation unit 25 requests the virtual display control unit 26 to generate a field of view screen including the now switching button 134 and the later switching button 135.

[0163] S66: The virtual display control unit 26 synthesizes the now switch button 134 and the later switch button 135 on the current view screen of the instructor avatar.

[0164] S67: The virtual display control unit 26 transmits screen information of a field of view screen on which the Now Switch button 134 and the Later Switch button 135 are combined to the VR goggles 5 via the communication control unit 23. The wireless LAN communication control unit 101 of the VR goggles 5 receives this screen information, and the display control unit 100 displays the field of view screen on which the Now Switch button 134 and the Later Switch button 135 are combined (see FIG. 15 ).

[0165] Next, a case where instructor Z presses the Switch Now button 134 on the field of view screen will be described. In step S70, instructor Z moves the pointer 170 to the Switch Now button 134. When the VR goggles 5 of the instructor that output the notification message 133 receives an operation to display field of view information for group 2, the virtual display control unit 26 generates field of view information for instructor avatar Z2 of instructor Z who is placed in group 2, and the communication control unit 23 transmits this information to the VR goggles 5 of instructor Z.

[0166] S71: Instructor Z points the pointer 170 to the switch button 134 now and presses the confirm button 116.

[0167] S72-1: The VR operation controller 6 accepts the pressing of the confirmation button 116, and the button information sending unit 121 sends to the VR goggles 5 via the Bluetooth communication control unit 120 a message that the switch now button 134 has been pressed. S72-2: The wireless LAN communication control unit 101 of the VR goggles 5 immediately transmits to the virtual conference server 1 a message that the switching button 134 has been pressed.

[0168] S73: The communication control unit 23 of the virtual conference server receives the information that the immediate switch button 134 has been pressed, and the virtual conference control unit 20 transmits a request to switch the field of view screen specifying group 2 to the virtual display control unit 26.

[0169] S74: In response to this request, the virtual display control unit 26 switches the field of view screen of the instructor avatar to the field of view screen of group 2. In detail, the field of view screen is an image projected in the direction in which instructor Z is looking in the virtual 360° virtual space in which group 2 is constructed.

[0170] S75: The virtual display control unit 26 transmits screen information of the field of view screen of group 2 to the VR goggles 5 via the communication control unit 23. The wireless LAN communication control unit 101 of the VR goggles 5 receives this screen information, and the display control unit 100 displays the field of view screen of group 2 (see FIG. 16).

[0171] Next, a case will be described in which instructor Z presses the "later" switch button 135 on the field of view screen of Fig. 15. In step S80, instructor Z moves the pointer 170 to the "later" switch button 135. When the VR goggles 5 of instructor Z displaying the notification message 133 accepts an operation to postpone responding to the notification message rather than an operation to display the field of view information of group 2, the output control unit 43 generates audio data indicating that the response will be made later. The communication control unit 23 transmits the audio data to the VR goggles 12 of group 2.

[0172] S81: Instructor Z points the pointer 170 to the "later" switch button 135 and presses the "confirm" button 116.

[0173] S82-1: The VR operation controller 6 accepts the pressing of the confirmation button 116, and the button information transmitting unit 121 later transmits to the VR goggles 5 via the Bluetooth communication control unit 120 a message that the switching button 135 has been pressed. S82-2: The wireless LAN communication control unit 101 of the VR goggles 5 later transmits to the virtual conference server 1 a message that the switching button 135 has been pressed.

[0174] S83: The communication control unit 23 of the virtual conference server later receives the information that the switching button 135 has been pressed, and the virtual conference control unit 20 sends a request to the display data generation unit 25 to generate a notification button 132 with a mark 136 added.

[0175] S84: In response to this request, the display data generating unit 25 generates the notification button 132 to which the mark 136 has been added.

[0176] S85: Furthermore, the display data generation unit 25 requests the virtual display control unit 26 to generate a field of view screen including the notification button 132 to which the mark 136 has been added.

[0177] S86: The virtual display control unit 26 combines the notification button 132 with the mark 136 added thereto onto the current view screen of the instructor avatar Z1.

[0178] S87: The virtual display control unit 26 transmits screen information of the field of view screen onto which the notification button 132 with the mark 136 added is composited to the VR goggles 5 via the communication control unit 23. The wireless LAN communication control unit 101 of the VR goggles 5 receives this screen information, and the display control unit 100 displays the field of view screen onto which the notification button 132 with the mark 136 added is composited (see FIG. 17 ).

[0179] S88: Furthermore, since the later switching button 135 is pressed, the virtual conference control unit 20 requests the output control unit 43 to utter the utterance "I will join the discussion later."

[0180] S89: In response to this request, the output control unit 43 generates voice data for the utterance, “I will join the discussion later.” The output control unit 43 also stores the notification message 133 in the conference data storage unit 24.

[0181] S90: The output control unit 43 transmits the audio data to the VR goggles 12 of each participant of group 2 via the communication control unit 23.

[0182] S91: The wireless LAN communication control unit 101 of the VR goggles 12 receives the voice data and outputs a voice message saying "I will join the discussion later" from the speaker 92. Note that although the VR goggles 12 outputs the voice message "I will join the discussion later," the same message may be displayed as linguistic information. In this case, a notification message is synthesized on the field of view screen of group 2.

[0183] <Major Effects> In the virtual space system 300 of this embodiment, in a situation where instructor Z participates in group work and places instructor avatars Z1 to Z3 that represent his / her avatar in each group, notifications according to the status of groups that instructor Z is not moderating are sent to instructor avatar Z1, so that instructor Z can understand the status of groups that are not currently displayed on the VR goggles 5 and for which no audio data is being output.

[0184] [Second embodiment] In this embodiment, a virtual conference server 1 is described in which a group that the participant is not moderating notifies the instructor avatar Z1 of the group 1 that the participant is moderating that there is an inquiry such as a question.

[0185] Figure 21 shows an example of a statement made by an avatar in a group. In Figure 21, suppose that avatar J (Mr. Chiba) from group 3 makes a statement 161 to instructor avatar Z3, saying, "Mr. Nakazawa, is what I explained correct?" The content of statement 161 in the speech bubble in Figure 21 is for explanatory purposes and does not need to be actually displayed.

[0186] The view screen of Group 1 is displayed on Instructor Z's VR goggles 5, and the audio of Group 1 is output from speaker 92, so Instructor Z does not notice what Group 3 is saying.

[0187] Therefore, the voice recognition unit 42 constantly recognizes the speech of each group and transmits the speech data that is the recognition result to the output control unit 43. "Nakazawa-san" included in the speech 161 is the name of instructor Z (it may be just the first or last name), and when the output control unit 43 detects that there has been a speech 161 that includes the name of instructor Z, it determines that there has been an inquiry for instructor Z. The output control unit 43 issues a request to the virtual display control unit 26 of the virtual conference server 1 to add a notification message indicating that there has been an inquiry to the field of view screen of group 1 of instructor Z. The virtual display control unit 26 receives the request to add the notification message and synthesizes the notification message on the field of view screen of group 1 of instructor Z.

[0188] FIG. 22 is an example of the field of view screen of the instructor avatar of Group 1 on which a notification message of an inquiry is displayed. A notification message 137 saying "We have a question for you. From Chiba, Group 3" is displayed at the top of the field of view screen. The notification message 137 contains "from Chiba, Group 3," which indicates that an inquiry has been made by Chiba-san of Group 3. Note that the voice recognition unit 42 identifies the speaker based on which VR goggles 15 the voice data is from, so it can determine the participant who said "Nakazawa-san." The notification message 137 saying "We have a question for you. From Chiba, Group 3" may be output by voice (the notification message 137 may or may not be displayed).

[0189] In addition, since avatar J's statement may include "Nakazawa-san" even if it is not an inquiry, it is preferable that the output control unit 43 analyzes the context to determine whether it is an inquiry and notify the instructor avatar Z1 if it is an inquiry.

[0190] In this way, instructor Z can confirm that there has been an inquiry from another group whose VR goggles 5 is not currently displaying a view screen or outputting audio, by the notification message 137 superimposed on the view screen currently displayed on the VR goggles 5.

[0191] Regarding the action taken by instructor Z in response to the notification message 137 in Figure 22, as in Figure 15 of the first embodiment, instructor Z can select the action of switching immediately or later by selecting the switch now button 134 or the switch later button 135.

[0192] Furthermore, it is also possible to simultaneously display multiple notification messages 133, 137 on the field of view screen, as shown in Fig. 23. In Fig. 23, the output control unit 43 displays notification message 133 indicating that the discussion in group 2 is stalled, and notification message 137 indicating that there is an inquiry from group 3.

[0193] When multiple notification messages 133, 137 are displayed simultaneously, the older one may be displayed at the top, or after a certain period of time, the message may be displayed in a highlighted color such as red or may blink.

[0194] <Action or Processing> 24 is a sequence diagram illustrating the process in which the output control unit 43 notifies the instructor avatar of an inquiry from a group that is not currently displaying a field of view screen or outputting audio data on the VR goggles 5. As a premise of FIG. 24, the instructor avatar Z1 is placed in group 1, and one or more participant avatars are placed in group 3. If a comment made by a participant in group 3 is about instructor Z, the virtual display control unit 26 generates a notification message indicating that there is an inquiry from the participant, and the communication control unit 23 sends this to the instructor's VR goggles 5 that are displaying the field of view information for group 1.

[0195] S101: For example, Chiba-san from Group 3 spoke.

[0196] S102: The microphone 91 of Chiba's VR goggles acquires the speech and converts it into voice data. The wireless LAN communication control unit 101 of the VR goggles 15 transmits the voice data to the virtual conference server.

[0197] S103: The communication control unit 23 of the virtual conference server receives the voice data and recognizes the voice (converts it into text) using the voice recognition unit 42. The voice recognition unit 42 notifies the output control unit 43 of the name of the group that made the comment and the name of the speaker.

[0198] The output control unit 43 determines whether the utterance is an inquiry to the instructor (whether the instructor's first or last name is included), and if the utterance is an inquiry to instructor Z, the following processing is performed. That is, if the participants in group 3 include the first or last name of instructor Z, the virtual display control unit 26 generates a notification message indicating that there is an inquiry from a participant in another group, and the communication control unit 23 sends this to the instructor's VR goggles 5 that are displaying the field of view information for group 1.

[0199] S104: The output control unit 43 requests the display data generation unit 25 to generate a notification message indicating that an inquiry has been received. The output control unit 43 also attaches the name of the group that made the comment and the name of the person who made the comment.

[0200] S105: In response to this request, the display data generating unit 25 generates a notification message indicating that an inquiry has been received.

[0201] S106: The display data generation unit 25 also requests the virtual display control unit 26 to generate a field of view screen on which a notification message indicating that there is an inquiry is synthesized. This field of view screen is the field of view screen of the group currently displayed on the VR goggles 5 of the instructor Z.

[0202] S107: The virtual display control unit 26 generates a view screen in which a notification message indicating that an inquiry has been received is combined with the view screen of the current instructor avatar.

[0203] S108: The virtual display control unit 26 transmits screen information of the field of view screen onto which a notification message indicating that an inquiry has been received to the VR goggles 5 via the communication control unit 23. The wireless LAN communication control unit 101 of the VR goggles 5 receives this screen information, and the display control unit 100 displays the field of view screen onto which a notification message indicating that an inquiry has been received (see FIG. 22).

[0204] <Major Effects> According to this embodiment, in addition to the effects of the first embodiment, one instructor Z places instructor avatars Z1 to Z3 in multiple groups, and even if an inquiry comes from a group that is not displaying the field of view screen on the VR goggles 5, the instructor Z can notice the inquiry and respond quickly.

[0205] [Third embodiment] In this embodiment, an explanation will be given of the alter ego switching of the instructor avatar by pressing the alter ego switching button 131. Alter ego switching refers to the instructor Z arbitrarily switching which group's field of view screen is displayed on the VR goggles 5.

[0206] 25 shows an example of the avatar switching button 131 displayed on the VR goggles 5 of instructor Z. Instructor Z moves the pointer 170 and presses the avatar switching button 131. This causes the virtual display control unit 26 to display a field of view screen that combines a list 141 of switchable instructor avatars, as shown in FIG.

[0207] FIG. 26 shows a view screen displaying a list 141 of instructor avatars. In the list 141 of instructor avatars, instructor avatar buttons 141a to 141c are displayed for all instructor avatars assigned to each group. The instructor avatar of the group whose view screen is currently displayed on the VR goggles 5 and whose audio data is currently being output is called the "master." The master is indicated by an icon 143 on the instructor avatar button 141a. The view screen and audio data of the group with icon 143 are being transmitted to instructor Z's VR goggles 5 (instructor Z is moderating). In addition, the instructor avatar buttons 141a to 141c display the names of the groups in which instructor avatars Z1 to Z3 are assigned. These are registered in the correspondence information in FIG. 11.

[0208] When instructor Z selects, for example, instructor avatar button 141b of group 2 from instructor avatar list 141, virtual display control unit 26 switches the field of view screen of instructor Z's VR goggles 5 to the field of view screen seen by instructor avatar Z2 of group 2.

[0209] <Action or Processing> FIG. 27 is a sequence diagram illustrating a process for switching the view screen displayed by the VR goggles 5 of the instructor Z.

[0210] S121: Instructor Z moves the pointer 170 to the avatar switch button 131 while pressing the pointer display button 115.

[0211] S121-2: While instructor Z is pressing the pointer display button 115, the Bluetooth communication control unit 120 of the VR operation controller 6 periodically (for example, every 100 milliseconds) transmits information regarding the movement of the pointer (information regarding the orientation and movement of the device) to the VR goggles 5. S122: The wireless LAN communication control unit 101 of the VR goggles 5 transmits a position calculation request based on information about the movement of the pointer to the virtual conference service.

[0212] S123: The communication control unit 23 of the virtual conference service receives the position calculation request. The pointer position calculation unit 27 of the virtual conference service calculates the display position of the pointer 170 based on the information on the orientation and movement of the VR operation controller 6.

[0213] S124: The pointer position calculation unit 27 requests the display data generation unit 25 to generate the pointer 170 together with the calculated position.

[0214] S125: In response to this request, the display data generating unit 25 generates the pointer 170 at the calculated position.

[0215] S126: The display data generation unit 25 also requests the virtual display control unit 26 to generate a field of view screen that combines the moved pointer 170. This field of view screen is the field of view screen of the group that is currently displayed on the VR goggles 5 of instructor Z.

[0216] S127: The virtual display control unit 26 generates a field of view screen in which the pointer 170 after movement is superimposed on the field of view screen of the current instructor avatar.

[0217] S128: The virtual display control unit 26 transmits screen information of the field of view screen onto which the moved pointer 170 is composited to the VR goggles 5 via the communication control unit 23. The wireless LAN communication control unit 101 of the VR goggles 5 receives this screen information, and the display control unit 100 displays the field of view screen onto which the moved pointer 170 is composited (see FIG. 25).

[0218] S129: Instructor Z presses the confirm button 116 with the pointer 170 overlapping the avatar switch button 131.

[0219] S130-1: The Bluetooth communication control unit 120 of the VR operation controller 6 transmits a confirmation command to the VR goggles 5 via Bluetooth. S130-2: The wireless LAN communication control unit 101 of the VR goggles 5 sends this command to the virtual conference service.

[0220] S131: When the communication control unit 23 of the virtual conference service receives this confirmation command, the virtual conference control unit 20 requests the display data generation unit 25 to generate a list 141 of instructor avatars.

[0221] S132: In response to this request, the display data generation unit 25 generates a list 141 of instructor avatars. The avatar management unit 44 manages the correspondence between groups 1 to 3 and instructor avatars Z1 to Z3, and also manages information about which group the instructor avatar is currently assigned to. The display data generation unit 25 obtains information about instructor avatars Z1 to Z3 assigned to each group from the avatar management unit 44.

[0222] S133: The display data generation unit 25 also requests the virtual display control unit 26 to generate a field of view screen that combines the list of instructor avatars 141. This field of view screen is the field of view screen of the group that is currently displayed on the VR goggles 5 of instructor Z.

[0223] S134: The virtual display control unit 26 generates a field of view screen in which the list 141 of instructor avatars is superimposed on the field of view screen of the current instructor avatar.

[0224] S135: The virtual display control unit 26 transmits screen information of the field of view screen onto which the list 141 of instructor avatars has been superimposed to the VR goggles 5 via the communication control unit 23. The wireless LAN communication control unit 101 of the VR goggles 5 receives this screen information, and the display control unit 100 displays the field of view screen onto which the list 141 of instructor avatars has been superimposed (see FIG. 26).

[0225] In step S136, it is assumed that instructor Z operates the VR operation controller 6 to move the pointer 170 to the list of instructor avatars 141. This process may be the same as steps S121 to S128.

[0226] S137: Instructor Z presses the confirm button 116 with the pointer 170 overlapping the instructor avatar button 141b. In this way, when the VR goggles 5 receives the selection of an arbitrary avatar from the list of instructor Z's avatars arranged in groups 1 to 3, the virtual display control unit 26 generates field of view information of the group in which the selected avatar is arranged, and the communication control unit 23 transmits the field of view information to the VR goggles 5 of instructor Z.

[0227] S138-1: The Bluetooth communication control unit 120 of the VR operation controller 6 transmits a confirmation command together with the identification information of the selected instructor avatar button 141b to the VR goggles 5 via Bluetooth. S138-2: The wireless LAN communication control unit 101 of the VR goggles 5 transmits this confirmation command to the virtual conference service.

[0228] S139: When the communication control unit 23 of the virtual conference service receives the identification information of the lecturer avatar button 141b and the confirmation command, the virtual conference control unit 20 requests the virtual display control unit 26 to specify a group and switch the field of view screen.

[0229] S140: The virtual display control unit 26 switches to the field of view screen of the specified group.

[0230] S141: The virtual display control unit 26 transmits screen information of the field of view screen of the switching destination group to the VR goggles 5 via the communication control unit 23. The wireless LAN communication control unit 101 of the VR goggles 5 receives this screen information, and the display control unit 100 displays the field of view screen of the switching destination group.

[0231] <Major Effects> According to this embodiment, when instructor avatars Z1 to Z3 of instructor Z are arranged in groups 1 to 3, instructor Z can switch the field of view screen of any group.

[0232] [Fourth embodiment] In this embodiment, re-display of the notification message by pressing the notification button 132 will be described.

[0233] 28 shows a view screen including the notification button 132 with the mark 136 added. When the instructor Z presses the notification button 132, the virtual display control unit 26 acquires the content of the notification message from the conference data storage unit 24, and generates a view screen with the notification message 133 superimposed thereon, as shown in FIG.

[0234] <Action or Processing> 29 is a sequence diagram illustrating the process in which instructor Z causes the VR goggles 5 to display the notification message again. When the VR operation controller 6 of instructor Z receives an operation to display the notification message, the communication control unit 23 transmits the notification message acquired from the conference data storage unit 24 to the VR goggles 5 of instructor Z that are displaying the field of view information of group 1.

[0235] S151: Instructor Z presses the confirm button 116 with the pointer 170 overlapping the notification button 132.

[0236] S152-1: The Bluetooth communication control unit 120 of the VR operation controller 6 transmits a confirmation command along with a notice that the notification button 132 has been pressed to the VR goggles 5 via Bluetooth. S152-2: The wireless LAN communication control unit 101 of the VR goggles 5 transmits this confirmation command to the virtual conference service.

[0237] S153: When the communication control unit 23 of the virtual conference service receives the confirmation command, the virtual conference control unit 20 requests the virtual display control unit 26 to re-display the notification message.

[0238] S154: In response to this request, the virtual display control unit 26 obtains from the conference data storage unit 24 a notification message indicating that the instructor Z has not pressed the switch button 134 now. The switch button 134 has not been pressed now means that the instructor Z is not responding to an inquiry from the group (the field of view screen for the group is not being displayed).

[0239] S155: The virtual display control unit 26 generates a view screen in which the notification message is combined with the view screen of the current instructor avatar.

[0240] S156: The virtual display control unit 26 transmits screen information of the field of view screen onto which the notification message has been combined to the VR goggles 5 via the communication control unit 23. The wireless LAN communication control unit 101 of the VR goggles 5 receives this screen information, and the display control unit 100 displays the field of view screen onto which the notification message has been combined.

[0241] FIG. 30 is an example of the field of view screen of the instructor avatar of group 1 on which the inquiry notification message 137 is displayed again. The explanation of FIG. 30 will mainly focus on the differences from FIG. 22. If instructor Z selects the switch button 135 later when the notification message 133 is displayed, the notification message 133 may also be displayed. In FIG. 30, the time when the notification message 137 was first displayed is displayed. This allows instructor Z to notice that time has passed since the first display and to respond promptly.

[0242] Furthermore, even if instructor Z does not press the notification button 132, the output control unit 43 may automatically display the notification message 137 when a certain time has passed since the first display time of the notification message 137. In this way, even if instructor Z forgets that he or she has not responded to the notification message 137, the output control unit 43 can remind him or her.

[0243] <Major Effects> According to this embodiment, the notification message regarding the group for which instructor Z has postponed handling can be displayed again.

[0244] [Other application examples] The best mode for carrying out the present invention has been described above using examples, but the present invention is not limited to these examples in any way, and various modifications and substitutions can be made within the scope that does not deviate from the gist of the present invention.

[0245] For example, in this embodiment, the VR goggles 5 of instructor Z switch and display the field of view screen of the group, but the field of view screen may also be displayed on the notebook PC 4 of instructor Z. Also, the field of view screen of instructor avatars Z1 to Z3 of each group may be displayed constantly on a large display such as an electronic whiteboard.

[0246] Furthermore, the content of the notification message sent when instructor Z is not moderating may be anything other than a stalled discussion or an inquiry. For example, the notification may be that the discussion is getting heated, or the output control unit 43 may detect and notify that the discussion is moving in a direction that is different from the correct answer.

[0247] Furthermore, in this embodiment, the VR goggles 5 of instructor Z output audio data for the group that displays the field of view screen, but the VR goggles 5 may be configured to display a different group for the field of view screen and output audio data. For example, the VR goggles 5 may be configured to display a field of view screen for group 1 but output audio data for group 2. Instructor Z can select the group that displays the field of view screen and the group that outputs audio data.

[0248] In this embodiment, when the VR goggles 5 of instructor Z displays a notification message and the switch now button 134 is pressed, the field of view screen of group 2 is displayed, but the VR goggles 5 may split the LCD 87 to simultaneously display the field of view screens of groups 1 and 2. In this case, it is conceivable to temporarily switch the audio data to group 2 or have instructor Z select it.

[0249] Furthermore, when the switching button 135 is pressed later, the field of view screen displayed by the VR goggles 5 may remain in group 1, and only the audio data may be switched to group 2 or 3. In this case, the VR goggles 5 may display on the LCD 87 which group's audio data is being output. Instructor Z may be able to arbitrarily select the group for which audio data is to be output.

[0250] Furthermore, in this embodiment, the participants operate the VR operation controller to select the virtual interactive whiteboard display data, but voice operation may also be used.

[0251] Although the present embodiment does not provide a detailed description of an actual electronic whiteboard, if an electronic whiteboard is present in the real space, the screen of the electronic whiteboard in the real space can be synchronized with the virtual conference space 8.

[0252] Furthermore, the configuration examples shown in Figure 5 and other figures are divided according to main functions to make it easier to understand the processing by the virtual space system 300. The method of dividing the processing units or the names of the processing units does not limit the present invention. The processing by the virtual space system 300 can also be divided into more processing units depending on the processing content. Furthermore, it can also be divided so that one processing unit includes more processes.

[0253] Furthermore, each function of the above-described embodiments can be realized by one or more processing circuits. Here, the term "processing circuit" in this specification includes a processor programmed to perform each function by software, such as a processor implemented by an electronic circuit, as well as devices such as an ASIC (Application Specific Integrated Circuit), a DSP (Digital Signal Processor), an FPGA (Field Programmable Gate Array), and conventional circuit modules designed to perform each of the above-described functions.

[0254] Embodiments of the present invention provide significant improvements in computer power and functionality. These improvements allow users to utilize computers that provide more efficient and robust interaction with tables, the way information is stored and presented in information processing devices. Furthermore, embodiments of the present invention provide a better user experience through the use of more efficient, powerful, and robust user interfaces. Such user interfaces provide better interaction between humans and machines. <Additional Notes> [Appendix 1] An information processing system that transmits visibility information of an avatar placed in a virtual space to a terminal device via a network, an avatar management unit that arranges a plurality of avatars of a first user in different virtual spaces; a virtual display control unit that generates information about the virtual space that is not displayed on the terminal device of the first user; a communication control unit that transmits information about the virtual space to the terminal device of the first user; An information processing system having the above. [Appendix 2] an avatar of the first user is placed in a first virtual space, and one or more avatars of second users are placed in a second virtual space; If no utterance of the second user is detected in the second virtual space, the virtual display control unit generates a message indicating that no utterance of the second user is detected in the second virtual space; The information processing system according to claim 1, wherein the communication control unit transmits the message to the terminal device of the first user that is displaying field of view information of the first virtual space. [Appendix 3] an avatar of the first user is placed in a first virtual space, and one or more avatars of second users are placed in a second virtual space; If the second user's comment in the second virtual space is related to the first user, the virtual display control unit generates a message indicating that there is an inquiry from the second user; The information processing system according to claim 1, wherein the communication control unit transmits the message to the terminal device of the first user that is displaying field of view information of the first virtual space. [Appendix 4] If the second user's statement in the second virtual space includes the first name or last name of the first user, the virtual display control unit generates a message indicating that there is an inquiry from the second user; The information processing system according to claim 3, wherein the communication control unit transmits the message to the terminal device of the first user that is displaying field of view information of the first virtual space. [Appendix 5] When the terminal device of the first user who output the message receives an operation to display view information of the second virtual space, the virtual display control unit generates visibility information of an avatar of the first user placed in the second virtual space; 5. The information processing system according to claim 2, wherein the communication control unit transmits the visibility information to the terminal device of the first user. [Appendix 6] When the terminal device of the first user that displayed the message receives an operation to postpone the response to the message instead of an operation to display the view information of the second virtual space, an output control unit that generates voice data indicating that the message will be addressed later; 5. The information processing system according to claim 2, wherein the communication control unit transmits the voice data to a terminal device of the second user. [Appendix 7] When the terminal device of the first user that displayed the message receives an operation to postpone the response to the message instead of an operation to display the view information of the second virtual space, The output control unit stores the message in a storage unit, When the terminal device of the first user receives an operation to display the message, An information processing system as described in Appendix 6, wherein the communication control unit transmits the message obtained from the memory unit to the terminal device of the first user displaying field of view information of the first virtual space. [Appendix 8] When a selection of an arbitrary avatar is received from a list of avatars of the first user placed in the virtual space displayed by the terminal device of the first user, the virtual display control unit generates view information of the virtual space in which the selected avatar is placed; 8. The information processing system according to claim 1, wherein the communication control unit transmits the visibility information to a terminal device of the first user. [Explanation of symbols]

[0255] 1 Virtual Conference Server 2. Electronic whiteboard 4,11,14 Laptop 5,12,15 VR goggles 6,13,16 VR operation controller 8 Virtual Meeting Spaces 9 Display Data [Prior art documents] [Patent documents]

[0256] [Patent Document 1] Japanese Patent Application Laid-Open No. 2009-145883

Claims

1. An information processing system that transmits visibility information of an avatar placed in a virtual space to a terminal device via a network, an avatar management unit that arranges a plurality of avatars of a first user in different virtual spaces; a virtual display control unit that generates information about the virtual space that is not displayed on the terminal device of the first user; a communication control unit that transmits information about the virtual space to the terminal device of the first user; An information processing system having the above.

2. an avatar of the first user is located in a first virtual space, and one or more avatars of second users are located in a second virtual space; If no utterance of the second user is detected in the second virtual space, the virtual display control unit generates a message indicating that no utterances of the second user are detected in the second virtual space; The information processing system according to claim 1 , wherein the communication control unit transmits the message to the terminal device of the first user that is displaying the field of view information of the first virtual space.

3. an avatar of the first user is located in a first virtual space, and one or more avatars of second users are located in a second virtual space; If the statement of the second user in the second virtual space is related to the first user, the virtual display control unit generates a message indicating that there is an inquiry from the second user; The information processing system according to claim 1 , wherein the communication control unit transmits the message to the terminal device of the first user that is displaying the field of view information of the first virtual space.

4. If the second user's comment in the second virtual space includes the first name or last name of the first user, the virtual display control unit generates a message indicating that there is an inquiry from the second user; The information processing system according to claim 3 , wherein the communication control unit transmits the message to the terminal device of the first user that is displaying the field of view information of the first virtual space.

5. When the terminal device of the first user who output the message receives an operation to display view information of the second virtual space, the virtual display control unit generates visibility information of an avatar of the first user arranged in the second virtual space; 5. The information processing system according to claim 2, wherein the communication control unit transmits the visibility information to the terminal device of the first user.

6. When the terminal device of the first user that displayed the message receives an operation to postpone the response to the message instead of an operation to display the view information of the second virtual space, an output control unit that generates voice data indicating that the message will be addressed later; 5. The information processing system according to claim 2, wherein the communication control unit transmits the voice data to the terminal device of the second user.

7. When the terminal device of the first user that displayed the message receives an operation to postpone the response to the message instead of an operation to display the view information of the second virtual space, The output control unit stores the message in a storage unit, When the terminal device of the first user receives an operation to display the message, The information processing system according to claim 6 , wherein the communication control unit transmits the message obtained from the memory unit to the terminal device of the first user that is displaying field of view information of the first virtual space.

8. When a selection of an arbitrary avatar is received from a list of avatars of the first user arranged in the virtual space displayed by the terminal device of the first user, the virtual display control unit generates view information of the virtual space in which the selected avatar is placed; The information processing system according to claim 1 , wherein the communication control unit transmits the visibility information to the terminal device of the first user.

9. An output method performed by an information processing system that transmits visual field information of an avatar placed in a virtual space to a terminal device via a network, comprising: placing a plurality of avatars of the first user in different virtual spaces; A process of generating information about the virtual space that is not displayed on the terminal device of the first user; transmitting information about the virtual space to the terminal device of the first user; Output method.

10. An information processing system that transmits visual field information of an avatar placed in a virtual space to a terminal device via a network, an avatar management unit that arranges a plurality of avatars of a first user in different virtual spaces; a virtual display control unit that generates information about the virtual space that is not displayed on the terminal device of the first user; a communication control unit that transmits information about the virtual space to the terminal device of the first user; A program to function as a

Images