Virtual space system, information processing system, method for synchronization, and electronic blackboard

Abstract

To make virtual electronic blackboard display data selected by a user from a plurality of pieces of virtual electronic blackboard display data displayed in a virtual space synchronize with a real electronic blackboard.SOLUTION: The present invention relates to a virtual space system for displaying display data displayed in a virtual space in a real electronic blackboard. The virtual space system includes a synchronization control unit for making the display data selected by the user synchronize with the image data displayed in the electronic blackboard in a case where the virtual space displays a plurality of pieces of the display data.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a virtual space system, an information processing system, a synchronization method, and an electronic whiteboard. [Background technology]

[0002] There is a known technology that displays a PC (Personal Computer) screen or an electronic whiteboard screen in a virtual space, and enables users to conduct tasks such as meetings while viewing the virtual space screen by wearing VR (Virtual Reality) goggles. Conventional online meetings require the display of an electronic whiteboard screen on the PC screen, but in the virtual space (within the field of view of the VR goggles), the PC screen, electronic whiteboard screen, etc. can each be displayed.

[0003] There is known a technology for linking an electronic whiteboard screen displayed in a virtual space with a real electronic whiteboard screen (see, for example, Patent Document 1). Patent Document 1 discloses a technology for reflecting handwritten drawing data on a real electronic whiteboard on the screen of an electronic whiteboard in a virtual space. Summary of the Invention [Problem to be solved by the invention]

[0004] However, in the conventional technology, when multiple virtual electronic whiteboard display data are displayed in a virtual space, the user cannot select the virtual electronic whiteboard display data with which writing on the real electronic whiteboard is synchronized. Even if there is only one real electronic whiteboard, multiple electronic whiteboards can be generated in the virtual space, so if the user cannot select which electronic whiteboard in the virtual space to synchronize with the real electronic whiteboard, various inconveniences will occur.

[0005] In view of the above-mentioned problems, the present invention provides a technique for synchronizing virtual electronic whiteboard display data selected by a user from a plurality of virtual electronic whiteboard display data displayed in a virtual space with a real electronic whiteboard. [Means for solving the problem]

[0006] In view of the above problems, the present invention provides a virtual space system that displays display data displayed in a virtual space on a real electronic whiteboard, and when multiple pieces of display data are displayed in the virtual space, the system has a synchronization control unit that synchronizes the display data selected by a user with the screen data displayed by the electronic whiteboard. Effect of the Invention

[0007] The virtual electronic whiteboard display data selected by the user from a plurality of virtual electronic whiteboard display data displayed in the virtual space can be synchronized with the real electronic whiteboard. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a system configuration diagram of an example of a virtual space system. [Diagram 2] FIG. 2 is a diagram illustrating a hardware configuration of an example of an electronic whiteboard. [Diagram 3] FIG. 2 is a hardware configuration diagram of an example of a virtual conference server and a notebook PC. [Figure 4] FIG. 2 is a hardware configuration diagram of an example of VR goggles. [Diagram 5] FIG. 1 is a hardware configuration diagram of an example of a VR operation controller. [Figure 6] FIG. 1 is a functional block diagram of an example that explains the functions of a virtual conference server, a notebook PC, an electronic whiteboard, VR goggles, and a VR operation controller by dividing them into blocks. [Figure 7] FIG. 11 is a sequence diagram illustrating an example of a process until user A starts a conference in a virtual conference space. [Figure 8] FIG. 11 is a sequence diagram illustrating an example of a process for user B to participate in a conference in a virtual conference space. [Figure 9] FIG. 13 is a diagram showing an example of a menu list screen displayed in the virtual conference space. [Figure 10] FIG. 13 is a diagram showing an example of notebook PC display data displayed in a virtual conference space. [Figure 11A] FIG. 11 is a sequence diagram of an example of an operation in which a user A moves a pointer to virtual electronic whiteboard display data in order to display the screen of the virtual electronic whiteboard display data on the electronic whiteboard (part 1). [Figure 11B] FIG. 2 is a sequence diagram of an example of an operation in which a user A moves a pointer to virtual electronic whiteboard display data in order to display the screen of the virtual electronic whiteboard display data on the electronic whiteboard (part 2). [Figure 12] FIG. 13 shows an example inquiry screen displayed in a virtual conference space that can be viewed using VR goggles. [Figure 13] FIG. 13 is a diagram showing an example of a display of an electronic whiteboard on which handwritten drawing data is displayed and a virtual conference space. [Figure 14] FIG. 13 is a diagram showing an example of virtual electronic whiteboard display data in which a frame is highlighted by a user hovering a pointer over the virtual electronic whiteboard; [Figure 15] FIG. 13 illustrates an example message displayed in a virtual meeting space. [Figure 16] FIG. 2 shows an example synchronization confirmation screen 250 displayed in the virtual conference space. [Figure 17] FIG. 2 is a functional configuration diagram of an example of a virtual conference server. [Figure 18] 11 is a sequence diagram illustrating an example of a process in which an electronic whiteboard displays a thumbnail image of virtual electronic whiteboard display data displayed in a virtual conference space. FIG. [Figure 19] FIG. 13 is a diagram showing an example of a virtual whiteboard list display button that the electronic whiteboard has. [Figure 20] FIG. 13 is a diagram showing a thumbnail image of an example of virtual electronic whiteboard display data displayed on the electronic whiteboard. [Figure 21] 11A and 11B are diagrams illustrating an operation of a user to switch virtual electronic whiteboard display data to be displayed on a display. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS A virtual space system and a synchronization method performed by the virtual space system will be described below with reference to the drawings as an example of an embodiment of the present invention.

[0010] <Terminology> An electronic whiteboard is a device that uses a touch panel to display handwritten characters and figures on a display in real time. Electronic whiteboards may have functions such as network connection and converting handwritten characters into font data using OCR.

[0011] Screen data is data that constitutes the screen displayed on the display (screen) of an electronic whiteboard or a PC. Screen data may be image data captured from the screen.

[0012] Synchronization refers to keeping data or files in the same state between two or more different devices. In this embodiment, synchronization refers to keeping the display data of the virtual electronic whiteboard and the screen data of the real electronic whiteboard in the same state with a practical delay (for example, real time).

[0013] Real-time means that a computer performs some processing in response to some change in a situation within an allowable time range that can be considered to be almost simultaneous with the change.

[0014] The virtual space is a virtual space that reproduces, for example, a 360° three-dimensional space by a computer. The virtual space is a place where people interact and engage in social activities on a network as in the real world, and is a term that likens it to a physical space. In this embodiment, a conference room is mainly reproduced by the virtual space, but the virtual space can reproduce any three-dimensional space. In this embodiment, the virtual space may be a two-dimensional space as long as planar display data is displayed.

[0015] The virtual electronic whiteboard display data refers to the screen data displayed by the electronic whiteboard, which is displayed in a virtual space. Therefore, the virtual electronic whiteboard display data is the same as the screen data displayed by the electronic whiteboard.

[0016] The operation information by the user may be information related to the operation of the VR operation controller, the electronic whiteboard, etc., and may be information related to the operation of the user selecting the virtual electronic whiteboard display data to be synchronized.

[0017] A pointer is a type of cursor that indicates the input position on a computer screen, for example an arrow-shaped icon. Pointers are usually arrow-shaped, but the shape of the cursor can change depending on the state of the computer or the position of the mouse pointer.

[0018] In this embodiment, the display device for allowing the user to view the virtual space is referred to as a VR goggle. The VR goggle may be an AR (Augmented Reality) goggle or an MR (Mixed Reality) goggle.

[0019] [First embodiment] <System configuration example> 1 is a system configuration diagram of a virtual space system 300. In the virtual space system 300, Wi-Fi routers 3 and 10 in a conference room 17 and a home 18, and an electronic whiteboard 2 are communicatively connected via a virtual conference server 1 and a network N. Notebook PCs 4, 11, and 14 can be communicatively connected to the Wi-Fi routers 3 and 10. The network N is, for example, the Internet, but may also be an on-premise network such as an in-house network.

[0020] The virtual conference server 1 is an information processing system having one or more information processing devices. The virtual conference server 1 stores data of the virtual conference space 8 in a memory and executes a virtual conference application. Hereinafter, the service provided by the virtual conference application executed by the virtual conference server 1 is referred to as a virtual conference service. The virtual conference space 8 is, for example, a 360° three-dimensional space that imitates a conference room in a virtual space. The VR goggles 5, 12, and 15 display any display data in the virtual space, and the participants of the conference can view any viewpoint of the virtual conference space 8 through the VR goggles 5, 12, and 15. The participants can view the screens of the devices placed in the conference room as if they were in the conference room.

[0021] The virtual conference space 8 has a screen of an electronic whiteboard in a virtual space (hereinafter referred to as a virtual electronic whiteboard 40) that is a screen acquired from the real electronic whiteboard 2 and synchronized with this screen. The virtual conference space 8 has a screen of a notebook PC in a virtual space (hereinafter referred to as a virtual notebook PC) that is a screen acquired from the notebook PCs 4, 11, and 14 and synchronized with this screen. In the virtual conference space 8, the screen data of the virtual electronic whiteboard 40 is called virtual electronic whiteboard display data 9c and 9d, and the screen data of the notebook PCs 4, 11, and 14 is called PC display data 9a and 9b. In FIG. 1, the virtual electronic whiteboard 40 and the virtual electronic whiteboard display data 9c and 9d are shown as the same thing. When the virtual electronic whiteboard display data 9c and 9d and the PC display data 9a and 9b are not distinguished from each other, the data displayed in the virtual conference space 8 is called display data 9.

[0022] An electronic whiteboard 2 and a Wi-Fi router 3 are installed in a conference room 17, and a user A is in the conference room 17 bringing with him a laptop PC 4, VR goggles 5, and a VR operation controller 6. The electronic whiteboard 2 and the Wi-Fi router 3 are connected to a network N. The laptop PC 4 and the VR goggles 5 communicate with the Wi-Fi router 3 via wireless LAN. The VR goggles 5 and the VR operation controller 6 are connected, for example, via Bluetooth (registered trademark; omitted below). Bluetooth may be wireless LAN. The laptop PC 4 and the VR goggles 5 are also connected via Bluetooth (or wireless LAN). The laptop PC 4 and the electronic whiteboard 2 are also connected via Bluetooth (or wireless LAN).

[0023] A user C is in a conference room with a laptop PC 14, VR goggles 15, and a VR operation controller 16. The laptop PC 14 and the VR goggles 15 communicate with a Wi-Fi router 3 via wireless LAN. The VR goggles 15 and the VR operation controller 16 are connected, for example, via Bluetooth (or wireless LAN). The laptop PC 14 and the VR goggles 15 are also connected via Bluetooth (or wireless LAN). The laptop PC 14 and the electronic whiteboard 2 are also connected via Bluetooth (or wireless LAN).

[0024] The electronic whiteboard 2 and the laptops 4 and 14 in the conference room 17 are each running a virtual meeting app. The virtual meeting app may be a native app or a web app.

[0025] On the other hand, user B participates in the conference from his / her home 18. A Wi-Fi router 10 is installed in the home 18. User B uses a notebook PC 11, VR goggles 12, and a VR operation controller 13. The Wi-Fi router 10 in user B's home 18 is connected to a network N so as to be able to communicate with the network N.

[0026] The laptop PC 11 and VR goggles 12 of user B communicate with the Wi-Fi router 10 via wireless LAN. The VR goggles 12 and the VR operation controller 13 are connected via Bluetooth (or wireless LAN). The laptop PC 11 and the VR goggles are also connected via Bluetooth (or wireless LAN). The laptop PC 11 of user B runs a virtual conference app.

[0027] The notebook PCs 4, 11, and 14 are examples of terminal devices operated by users. The notebook PCs 4, 11, and 14 may be information processing devices that have a communication function and run a virtual conference application, such as a simple PC, a tablet terminal, a smartphone, or a PDA (Personal Digital Assistant).

[0028] 1, even if there is only one real electronic whiteboard 2, the user can generate multiple virtual electronic whiteboards in the virtual conference space 8. If the user cannot select which virtual electronic whiteboard in the virtual space to synchronize with the real electronic whiteboard 2, there is a risk that drawing data handwritten on the electronic whiteboard 2 will be reflected in a virtual electronic whiteboard that is not intended by the user. In addition, the virtual electronic whiteboard display data to be synchronized with the electronic whiteboard 2 will be fixed.

[0029] <Hardware configuration example> <<Electronic whiteboard>> Fig. 2 is a hardware configuration diagram of the electronic whiteboard 2. As shown in Fig. 2, the electronic whiteboard 2 includes a CPU (Central Processing Unit) 401, a ROM (Read Only Memory) 402, a RAM (Random Access Memory) 403, an SSD (Solid State Drive) 404, a network I / F 405, and an external device connection I / F (Interface) 406.

[0030] Of these, the CPU 401 controls the overall operation of the electronic whiteboard 2. The ROM 402 stores programs used to start the OS, such as an IPL (Initial Program Loader). The RAM 403 is used as a work area for the CPU 401. The SSD 404 stores various data, such as programs for the electronic whiteboard 2. The network I / F 405 controls communication with a communication network. The external device connection I / F 406 is an interface for connecting various external devices. In this case, the external devices are, for example, a USB (Universal Serial Bus) memory 430 and external devices (a microphone 440, a speaker 450, and a camera 460).

[0031] The electronic whiteboard 2 also includes a capture device 411, a GPU 412, a display controller 413, a contact sensor 414, a sensor controller 415, an electronic pen controller 416, a short-range communication circuit 419, an antenna 419a of the short-range communication circuit 419, a power switch 422, and selection switches 423.

[0032] Among these, the capture device 411 displays display information on the display of the external PC 470 as a still image or a moving image. The GPU (Graphics Processing Unit) 412 is a semiconductor chip that specializes in graphics. The display controller 413 controls and manages the screen display to output the output image from the GPU 412 to the display 480 or the like. The contact sensor 414 detects that the electronic pen 490, the user's hand H, or the like has contacted the display 480. The sensor controller 415 controls the processing of the contact sensor 414. The contact sensor 414 inputs and detects coordinates by an infrared blocking method. The method of inputting and detecting the coordinates is a method in which two light receiving and emitting devices installed at both ends of the upper side of the display 480 emit a plurality of infrared rays in parallel to the display 480, and receive the light that is reflected by a reflecting member installed around the display 480 and returns along the same optical path as the optical path of the light emitted by the light receiving element. The light receiving and emitting device, which is the contact sensor 414, outputs the position (position on the light receiving element) where the infrared ray is blocked by the object to the sensor controller 415, and the sensor controller 415 specifies the coordinate position, which is the contact position of the object, from the infrared blocking position information on these two light receiving elements. The electronic pen controller 416 determines whether the pen tip or the pen tail touches the display 480 by communicating with the electronic pen 490 via Bluetooth. The short-range communication circuit 419 is a communication circuit such as NFC (Near Field Communication) or Bluetooth (registered trademark). The power switch 422 is a switch for switching ON / OFF the power of the electronic whiteboard 2. The selection switches 423 are, for example, a group of switches for adjusting the brightness and color of the display 480.

[0033] Furthermore, the electronic whiteboard 2 includes a bus line 410. The bus line 410 is an address bus, a data bus, or the like for electrically connecting the components such as the CPU 401 shown in FIG.

[0034] The contact sensor 414 is not limited to an infrared blocking type, and may be a capacitive touch panel that identifies the contact position by detecting a change in capacitance. The contact sensor 414 may be a resistive film type touch panel that identifies the contact position by a voltage change in two opposing resistive films. The contact sensor 414 may be an electromagnetic induction type touch panel that identifies the contact position by detecting electromagnetic induction caused by a contact object touching the display unit. In addition, the contact sensor 414 may use various detection means. The electronic pen controller 416 may determine whether or not a touch has been made on not only the tip and end of the electronic pen 490 but also on a part of the electronic pen 490 that is held by the user or on other parts of the electronic pen.

[0035] <<Virtual conference server and laptop>> Fig. 3 is a diagram showing an example of a hardware configuration of the virtual conference server 1 and the notebook PCs 4, 11, and 14. As shown in Fig. 3, the virtual conference server 1 and the notebook PCs 4, 11, and 14 are constructed by a computer 500, and include a CPU 501, a ROM 502, a RAM 503, a HD (Hard Disk) 504, a HDD (Hard Disk Drive) controller 505, a display 506, an external device connection I / F (Interface) 508, a network I / F 509, a bus line 510, a keyboard 511, a pointing device 512, an optical drive 514, and a media I / F 516.

[0036] Among these, the CPU 501 controls the overall operation of the virtual conference server 1 and the notebook PCs 4, 11, and 14. The ROM 502 stores programs such as an IPL used to drive the CPU 501. The RAM 503 is used as a work area for the CPU 501. The HD 504 stores various data such as programs. The HDD controller 505 controls the reading or writing of various data from the HD 504 under the control of the CPU 501. The display 506 displays various information such as a cursor, a menu, a window, a character, or an image. The external device connection I / F 508 is an interface for connecting various external devices. In this case, the external device is, for example, a USB (Universal Serial Bus) memory or a printer. The network I / F 509 is an interface for data communication using a network. The bus line 510 is an address bus, a data bus, or the like for electrically connecting each component such as the CPU 501 shown in FIG. 3.

[0037] Moreover, the keyboard 511 is a type of input means having a plurality of keys used for inputting characters, numbers, various instructions, etc. The pointing device 512 is a type of input means for selecting and executing various instructions, selecting a processing target, moving a cursor, etc. The optical drive 514 controls reading and writing of various data from an optical storage medium 513, which is an example of a removable storage medium. The optical storage medium 513 may be a CD, a DVD, Blu-ray (registered trademark), etc. The media I / F 516 controls reading and writing (storing) of data from a storage medium 515 such as a flash memory.

[0038] <<VRゴーグル> > 4 is a hardware configuration diagram of the VR goggles 5, 12, and 15. The VR goggles 5, 12, and 15 each include a CPU 80, a main memory 81, a ROM 82, a display controller 86, a wireless LAN controller 88, an audio codec 90, and a video codec 93, which are connected via a bus 94.

[0039] The CPU 80 executes and processes the OS and control processing programs that are read from the ROM 82 to the main memory 81. The main memory 81 includes a DRAM, and is used as a work area for the CPU 80, and the like.

[0040] The ROM 82 is previously written with the OS, a system program for when the power is turned on, and a program for controlling the VR goggles 5, 12, and 15.

[0041] The UART 83 is connected to the CPU 80. The UART 83 is an interface for transmitting and receiving serial data between the CPU 80 and the Bluetooth module 84, and is composed of a FIFO (First In, First Out), a shift register, and the like.

[0042] The Bluetooth module 84 is composed of an RF section and a baseband section, and is connected to an antenna 85, and performs wireless communication in accordance with the Bluetooth standard.

[0043] A display controller 86 performs D / A conversion of character data, graphic data, image data, etc., and also controls the display of this data on an LCD 87 .

[0044] The wireless LAN controller 88 executes a communication protocol compliant with the IEEE (Institute of Electrical and Electronics Engineers) 802.11ax standard, and controls communication with other devices by transmitting and receiving radio waves via an antenna 89.

[0045] The audio signal input from the microphone 91 is converted into audio data by an A / D (Analog / Digital) conversion circuit, and this audio data is encoded by the audio codec 90 in the AAC (Advanced Audio Coding) format. Also, the AAC encoded data received from an external device is decoded by the audio codec 90, and the audio converted to analog by the D / A conversion circuit is output from the speaker 92. The video codec 93 decodes the video compression data (e.g., a format compliant with ITU-T Recommendation H.264) received from an external device. Data transfer between the above components is performed via the bus 94.

[0046] <<VR Operation Controller>> Figure 5 is a hardware configuration diagram of the VR operation controllers 6, 13, and 16. The VR operation controllers 6, 13, and 16 include a UART 117, a main memory 111, a ROM 112, a 6-axis acceleration / angular velocity sensor 113, a menu display button 114, a pointer display button 115, and an enter button 116, which are connected to the CPU 110.

[0047] The CPU 110 executes and processes the control processing program read from the ROM 112 into the main memory 111. The main memory 111 has a DRAM and is used as a work area of the CPU 110 and the like.

[0048] The ROM 112 is pre-written with a system program at power-on, and a program for transmitting the press information of the menu display button 114, the pointer display button 115, and the enter button 116 via Bluetooth.

[0049] The six-axis acceleration / angular velocity sensor 113 outputs measurement data of acceleration and angular velocity. The UART 117 is an interface for sending and receiving serial data between the CPU 110 and the Bluetooth module 118, and is composed of a FIFO, a shift register, etc. The Bluetooth module 118 is composed of an RF section and a baseband section, and is connected to an antenna 119, and performs wireless communication in accordance with the Bluetooth standard.

[0050] <About the function> Next, the functional configuration of the virtual space system 300 will be described with reference to Fig. 6. Fig. 6 is an example of a functional block diagram that explains the functions of the virtual conference server 1, the notebook PCs 4, 11, and 14, the electronic whiteboard 2, the VR goggles 5, 12, and 15, and the VR operation controllers 6, 13, and 16 in the virtual space system 300 by dividing them into blocks.

[0051] <<Virtual Conference Server>> The virtual conference server 1 has a virtual conference control unit 20, a whiteboard generation and management unit 21, a synchronization control unit 22, a communication control unit 23, a conference data storage unit 24, a display data generation unit 25, a virtual display control unit 26, a pointer position calculation unit 27, a user authentication unit 28, and a user information storage unit 29. Each of these units of the virtual conference server 1 is a function or a means for performing a function that is realized when any of the components shown in Fig. 3 operates in response to an instruction from the CPU 501 in accordance with a program loaded from the HD 504 to the RAM 503.

[0052] The virtual conference control unit 20 associates the conference participants and the devices connected to the conference (notebook PCs 4, 11, 14, electronic whiteboard 2, VR goggles 5, 12, 15) with the conference identification information (conference ID) and stores them in the conference data storage unit 24. The virtual conference control unit 20 executes a remote conference with each device for each conference ID. In addition, the virtual conference control unit 20 stores the screen data of the virtual electronic whiteboard, the audio data received from the devices connected to the virtual conference, and the like in the conference data storage unit 24.

[0053] The whiteboard generating and managing unit 21 generates a virtual electronic whiteboard in the virtual conference space 8, and manages the virtual electronic whiteboard display data 9c, 9d and the PC display data 9a, 9b.

[0054] The synchronization control unit 22 is a means for synchronizing the screen data of the electronic whiteboard 2 and the virtual electronic whiteboard display data 9c, 9d so that they are the same. Specifically, the synchronization control unit 22 includes a function for detecting updates of the screen data of the electronic whiteboard 2 and the virtual electronic whiteboard display data 9c, 9d, a function for reflecting the updated screen data of the electronic whiteboard 2 in the virtual electronic whiteboard display data 9c, 9d, and a function for reflecting the updated virtual electronic whiteboard display data 9c, 9d in the screen data of the electronic whiteboard 2. The synchronization control unit 22 also accepts selection of any virtual electronic whiteboard display data from a plurality of virtual electronic whiteboard display data displayed in the virtual conference space 8. The synchronization control unit 22 also accepts selection of virtual electronic whiteboard display data displayed on the electronic whiteboard 2 and selected by the user.

[0055] The communication control unit 23 transmits and receives data between the notebook PCs 4, 11, and 14, the VR goggles 5, 12, and 15, and the electronic whiteboard 2.

[0056] The conference data storage unit 24 stores the virtual electronic whiteboard display data 9c and 9d, the PC display data 9a and 9b, and audio data received from devices connected to the virtual conference.

[0057] The display data generating unit 25 generates the menu displayed in the virtual conference space 8, the pointer 170 instructed to be displayed by the VR operation controllers 6, 13, and 16, and the like.

[0058] The virtual display control unit 26 is a means for displaying a pointer 170 indicating virtual electronic whiteboard display data in the virtual conference space 8 based on operation information transmitted from the VR goggles 5, 12, and 15. Specifically, the virtual display control unit 26 displays various display data in the virtual conference space 8. The virtual display control unit 26 displays, in the virtual conference space 8, the menu and pointer 170 generated by the display data generation unit 25, the virtual electronic whiteboard display data 9c and 9d related to the virtual electronic whiteboard generated by the whiteboard generation and management unit 21, and the PC display data 9a and 9b related to the notebook PCs 4, 11, and 14. The virtual display control unit 26 transmits the display data of the virtual conference space 8 to the VR goggles 5, 12, and 15 via the communication control unit 23, and the virtual conference space 8 is displayed on the VR goggles 5, 12, and 15.

[0059] The pointer position calculation unit 27 calculates the position of the pointer 170 (in the virtual conference space 8) indicated by the orientation and movement of the VR operation controllers 6, 13, and 16.

[0060] The user authentication unit 28 compares the user name and password received via the communication control unit 23 with the user name and password stored in the user information storage unit 29, and authenticates the user participating in the virtual conference.

[0061] The user information storage unit 29 stores the user names and passwords of users who will participate in the virtual conference.

[0062] <<Notebook PC>> The notebook PCs 4, 11, and 14 have a virtual conference connection unit 30, a remote desktop control unit 31, a wireless LAN communication control unit 32, a Bluetooth communication control unit 33, a display control unit 34, and a data storage unit 35. Each of these units in the notebook PCs 4, 11, and 14 is a function or a means for performing the function, which is realized when any of the components shown in Fig. 3 operates in response to an instruction from the CPU 501 in accordance with the program (virtual conference application) loaded from the HD 504 to the RAM 503.

[0063] The virtual conference connection unit 30 connects to the virtual conference service of the virtual conference server 1 and transmits a user name and a password, which are authentication data. The virtual conference connection unit 30 also manages the URL and conference ID of the virtual conference space 8 received from the virtual conference service, and connects to the URL of the virtual conference space 8 to participate in the virtual conference. Furthermore, the virtual conference connection unit 30 transmits the URL and conference ID of the virtual conference space 8 to the VR goggles 5, 12, 15 and the electronic whiteboard 2 via the Bluetooth communication control unit 33.

[0064] The remote desktop control unit 31 transmits screen data of the notebook PCs 4 , 11 , and 14 to the virtual conference service to display the data in the virtual conference space 8 .

[0065] The wireless LAN communication control unit 32 executes communication with the virtual conference server 1 via the Wi-Fi routers 3 and 10 .

[0066] The Bluetooth communication control unit 33 executes communication in compliance with the Bluetooth standard with the VR goggles 5, 12, and 15 and the electronic whiteboard 2. The display control unit 34 performs control for displaying screen data on a display (LCD).

[0067] The data storage unit 35 stores the address (URL) of the virtual conference service provided by the virtual conference server 1, the URL of the virtual conference space 8 received from the virtual conference service, the conference ID, the user name and password of the user using the notebook PC 4, 11, 14, etc.

[0068] <<Electronic whiteboard>> The electronic whiteboard 2 has a contact position detection unit 70, a display control unit 71, a LAN communication control unit 72, a Bluetooth communication control unit 73, a drawing data generation unit 74, a virtual conference connection unit 75, and a data storage unit 76. Each of these units of the electronic whiteboard 2 is a function or a means for performing a function that is realized when any of the components shown in FIG. 2 operates in response to an instruction from the CPU 401 in accordance with the program (virtual conference application) expanded from the SSD 404 to the RAM 403.

[0069] When a finger, electronic pen 490, or the like comes into contact with display 480, contact position detection unit 70 detects coordinate data of a portion detected by contact sensor 414 (a portion where light is blocked).

[0070] The display control unit 71 is a unit that displays the virtual electronic whiteboard display data received by the LAN communication control unit 72 on the display 480. Specifically, the display control unit 71 performs control for displaying the screen data of the electronic whiteboard 2 on the display 480.

[0071] The LAN communication control unit 72 is a means for receiving virtual electronic whiteboard display data in order to synchronize the virtual electronic whiteboard display data selected by the user with the screen data displayed by the electronic whiteboard when multiple pieces of virtual electronic whiteboard display data are displayed in the virtual conference space 8. Specifically, the LAN communication control unit 72 connects to a network such as a Wi-Fi router 3 or Ethernet (registered trademark) and transmits and receives data to and from other devices via the network N.

[0072] The Bluetooth communication control unit 73 executes communication in compliance with the Bluetooth standard with the notebook PCs 4, 11, and 14. The drawing data generation unit 74 generates drawing data from a coordinate sequence input from the contact position detection unit 70 when the user writes by hand, and generates shapes such as circles and rectangles.

[0073] The virtual conference connection unit 75 uses the URL and conference ID of the virtual conference service transmitted from the notebook PC 4 to perform control for connecting to the virtual conference service executed by the virtual conference server 1 .

[0074] The data storage unit 76 stores the address (URL) and conference ID of the virtual conference service provided by the virtual conference server 1, the screen data displayed on the display 480, and the like.

[0075] <<VRゴーグル> > The VR goggles 5, 12, and 15 each have a display control unit 100, a wireless LAN communication control unit 101, a Bluetooth communication control unit 102, a virtual conference connection unit 103, a button information transfer unit 104, and a data storage unit 105. Each of these units in the VR goggles 5, 12, and 15 is a function or a means for performing a function that is realized when any of the components shown in Fig. 4 operates according to an instruction from the CPU 80 in accordance with a program (virtual conference application) loaded from the ROM 82 to the main memory 81.

[0076] The display control unit 100 performs control for displaying display data transmitted from the virtual conference service on the LCD 87. The wireless LAN communication control unit 101 executes communication with the virtual conference server 1 via the Wi-Fi routers 3 and 10.

[0077] The Bluetooth communication control unit 102 performs communication compliant with the Bluetooth standard with the VR operation controllers 6, 13, and 16. The virtual conference connection unit 103 performs control for connecting to the virtual conference service executed on the virtual conference server 1 using the URL and conference ID of the virtual conference service transmitted from the notebook PC 4.

[0078] The button information transfer unit 104 receives button press information from the VR operation controllers 6, 13, and 16, and transmits information regarding the pressed button to the virtual conference server 1 via the wireless LAN communication control unit 101.

[0079] The data storage unit 105 stores the address (URL) of the virtual conference service provided by the virtual conference server 1, the conference ID, the user name of the user using the VR goggles 5, 12, and the password, etc.

[0080] <<VR Operation Controller>> The VR operation controllers 6, 13, and 16 have a Bluetooth communication control unit 120, a button information transmission unit 121, and an orientation / movement detection unit 122. These functions of the VR operation controllers 6, 13, and 16 are functions or means realized by operating according to instructions from the CPU 110 based on a program expanded from the ROM 112 shown in FIG. 5 to the main memory 111.

[0081] The Bluetooth communication control unit 120 performs communication compliant with the Bluetooth standard with the VR goggles 5, 12, and 15. The button information transmission unit 121 passes the press information of the menu display button 114, the pointer display button 115, and the confirmation button 116 to the Bluetooth communication control unit 120.

[0082] The orientation / movement detection unit 122 detects the orientation (posture in three-dimensional space) and movement of the VR operation controllers 6, 13, and 16 from the acceleration and angular velocity data obtained from the 6-axis acceleration / angular velocity sensor 113, and passes this information to the Bluetooth communication control unit 120.

[0083] <Action or Processing> Next, referring to FIG. 7, a procedure will be described in which users A and B use the notebook PCs 4, 11 and VR goggles 5, 12 to join the virtual conference space 8 in the virtual conference server 1 and start a conference in the virtual space.

[0084] First, when user A enters the conference room 17 carrying the laptop PC 4, VR goggles 5, and VR operation controller 6, the laptop PC 4 detects the presence of the electronic whiteboard 2 via Bluetooth. The Bluetooth device periodically checks whether there is a Bluetooth device nearby (within range of Bluetooth radio waves). When the laptop PC 4 detects a Bluetooth device, it detects that the device is the electronic whiteboard 2 based on the product ID included in the device identification profile.

[0085] When User A, the conference organizer, starts the virtual conference app on the laptop PC 4, the virtual conference connection unit 30 connects to the virtual conference service of the virtual conference server 1 and displays a screen for inputting a user name and password. In the following, no distinction is made between the virtual conference server 1 and the virtual conference service, and the explanation is mainly given using the virtual conference service. The URL of the virtual conference service is registered in advance in the laptop PC 4 as setting data for the virtual conference app.

[0086] When User A inputs the user name and password, the virtual conference connection unit 30 transmits the user name and password to the virtual conference service via the wireless LAN communication control unit 32. The virtual conference service authenticates the user and transmits a response indicating that the authentication is OK to the notebook PC 4.

[0087] FIG. 7 is a sequence diagram illustrating the process up to when user A starts a conference in the virtual conference space 8. As shown in FIG.

[0088] S1: When the virtual conference connection unit 30 receives an authentication OK response via the wireless LAN communication control unit 32, the display control unit 34 of the virtual conference application running on the notebook PC 4 displays a start button and a join button for the virtual conference. User A is the organizer, so he selects the start button.

[0089] S2: The virtual conference connection unit 30 transmits a command to start the virtual conference to the virtual conference service via the wireless LAN communication control unit 32. This transmission uses, for example, HTTP.

[0090] S3: When the virtual conference control unit 20 of the virtual conference service receives this start command via the communication control unit 23, it generates a virtual conference space 8 and a conference ID.

[0091] S4: The communication control unit 23 transmits the URL of the virtual conference space 8 and the conference ID to the notebook PC 4.

[0092] S5: When the wireless LAN communication control unit 32 of the notebook PC 4 receives this data, the virtual conference connection unit 30 connects to the URL of the virtual conference space 8 via the wireless LAN communication control unit 32 and sends a conference join command including the received conference ID.

[0093] S6: The virtual conference control unit 20 of the virtual conference service compares this conference ID with the conference ID generated in step S3, and if it confirms a match, allows a connection between the notebook PC 4 and the virtual conference space 8 (participation in the virtual conference corresponding to the conference ID).

[0094] S7: The virtual conference control unit 20 transmits a response OK to the notebook PC 4 via the communication control unit 23.

[0095] S8: When the wireless LAN communication control unit 32 of the notebook PC 4 receives the response OK, the virtual conference connection unit 30 transmits the URL and conference ID of the virtual conference space 8 to user A's VR goggles 5 via Bluetooth via the Bluetooth communication control unit 33.

[0096] S9: When the Bluetooth communication control unit 102 of the VR goggles 5 receives this data, the virtual conference connection unit 103 connects to the URL of the virtual conference space 8 via the wireless LAN communication control unit 101 and sends a conference join command including the received conference ID.

[0097] S10: The virtual conference control unit 20 of the virtual conference service compares this conference ID with the conference ID generated in step S3, and if it confirms a match, allows a connection between the VR goggles 5 and the virtual conference space 8 (participation in the virtual conference corresponding to the conference ID).

[0098] S11: The virtual conference control unit 20 transmits a response OK to the VR goggles 5 via the communication control unit 23.

[0099] S11-2: The virtual display control unit 26 of the virtual conference service transmits the display data of the virtual conference space 8 to the VR goggles 5 via the communication control unit 23.

[0100] S11-3: When the wireless LAN communication control unit 101 of the VR goggles 5 receives this data, the display control unit 100 displays the received display data of the virtual conference space 8 on the LCD 87.

[0101] S12: The notebook PC 4 sends the URL and the conference ID of the virtual conference space 8 to the VR goggles 5, and then the virtual conference connection unit 30 sends the URL and the conference ID of the virtual conference space 8 to the electronic whiteboard 2 via Bluetooth via the Bluetooth communication control unit 33.

[0102] S13: When the Bluetooth communication control unit 73 of the electronic whiteboard 2 receives this data, the virtual conference connection unit 75 connects to the URL of the virtual conference space 8 via the LAN communication control unit 72 and sends a conference participation command including the received conference ID.

[0103] S14: The virtual conference control unit 20 of the virtual conference service compares this conference ID with the conference ID generated in step S3, and if there is a match, allows a connection between the electronic whiteboard 2 and the virtual conference space 8 (participation in the virtual conference corresponding to the conference ID).

[0104] S15: The virtual conference control unit 20 transmits a response OK to the electronic whiteboard 2 via the communication control unit 23.

[0105] S16: After the notebook PC 4 transmits the URL and the conference ID of the virtual conference space 8 to the electronic whiteboard 2, the display control unit 34 of the virtual conference application displays a selection button for the remote desktop.

[0106] S17: User A presses the selection button on the remote desktop.

[0107] S18: The remote desktop control unit 31 transmits the screen data of the notebook PC 4 to the virtual conference service via the wireless LAN communication control unit 32.

[0108] S19: The virtual display control unit 26 of the virtual conference service displays this screen data received by the communication control unit 23 in the virtual conference space 8 (PC display data 9a).

[0109] S20-1: The virtual display control unit 26 of the virtual conference service transmits the PC display data 9a received via the communication control unit 23 to the VR goggles 5.

[0110] S20-2: The display control unit 100 of the VR goggles 5 causes the LCD 87 to display this PC display data 9a.

[0111] With the above, user A has participated in the virtual conference, and the screen data of the notebook PC 4 is displayed on the VR goggles 5. Note that the virtual display control unit 26 of the virtual conference service controls so that the screen data of the notebook PC 4 displayed in the virtual conference space 8 is displayed only on the VR goggles 5 of user A.

[0112] <<User B joins virtual meeting>> Next, user A sends the URL of the virtual conference space 8 and the conference ID from the notebook PC 4 to the notebook PCs of the other conference participants by email or instant message. When user B, a conference participant, starts the virtual conference app on the notebook PC 11, the virtual conference connection unit 30 connects to the virtual conference service and displays a screen for inputting a user name and password. The URL of the virtual conference service is registered in advance in the notebook PC 11 as setting data for the virtual conference app.

[0113] When User B enters a user name and password, the virtual conference connection unit 30 transmits this data to the virtual conference service. The virtual conference service authenticates this user and transmits an authentication OK response to the notebook PC 11. The virtual conference application of the notebook PC 11 displays a button to start the virtual conference and a button to join the virtual conference.

[0114] FIG. 8 is a sequence diagram illustrating the process up to when user B participates in the conference in the virtual conference space 8. As shown in FIG.

[0115] S21: User B is a participant and therefore selects the join button.

[0116] S22: The display control unit 34 of the notebook PC 11 displays a screen for inputting the URL of the virtual conference space 8 and the conference ID.

[0117] S23: User B inputs the URL and conference ID of the virtual conference space 8 notified by User A, and presses the OK button.

[0118] S24: The virtual conference connection unit 30 transmits a connection command including this conference ID to the URL of the virtual conference space 8 via the wireless LAN communication control unit 32.

[0119] S25: The virtual conference control unit 20 of the virtual conference service collates this conference ID with the conference ID generated in step S3, and if they match, permits the connection between the notebook PC 11 and the virtual conference space 8.

[0120] S26: The virtual conference control unit 20 transmits a response OK to the notebook PC 11 via the communication control unit 23.

[0121] S27: Next, the Bluetooth communication control unit 33 of the notebook PC 11 transmits the URL of the virtual conference space 8 and the conference ID to the VR goggles 12 of the user B via Bluetooth.

[0122] S28: When the Bluetooth communication control unit 102 of the VR goggles 12 receives this data, the virtual conference connection unit 103 transmits a connection command including the conference ID to the URL of the virtual conference space 8 via the wireless LAN communication control unit 101.

[0123] S29: The virtual conference control unit 20 of the virtual conference service collates this conference ID with the conference ID generated in step S3, and if they match, permits connection between the VR goggles 12 and the virtual conference space 8.

[0124] S30: The virtual conference control unit 20 transmits a response OK to the VR goggles 12 via the communication control unit 23.

[0125] S30-2: The virtual display control unit 26 of the virtual conference service transmits display data of the virtual conference space 8 to the VR goggles 12 via the communication control unit 23.

[0126] S30-3: When the wireless LAN communication control unit 101 of the VR goggles 12 receives this data, the display control unit 100 displays the display data of the virtual conference space 8 received on the LCD 87.

[0127] S31: When the virtual conference connection unit 30 of the virtual conference application operating on the notebook PC 11 connects to the virtual conference space 8, the display control unit 34 displays the selection button for the remote desktop.

[0128] S32: User B selects the selection button for the remote desktop.

[0129] S33: The remote desktop control unit 31 of the notebook PC 11 transmits the screen data of the notebook PC 11 to the virtual conference service via the wireless LAN communication control unit 32.

[0130] S34: The virtual display control unit 26 of the virtual conference service displays the screen data of the notebook PC 11 in the virtual conference space 8 (PC display data 9b).

[0131] S35: The virtual display control unit 26 of the virtual conference service transmits the PC display data 9b to the VR goggles 12.

[0132] S36: The display control unit 100 of the VR goggles 12 causes the PC display data 9b to be displayed on the LCD 87.

[0133] Note that the virtual display control unit 26 of the virtual conference service controls so that the screen data of the notebook PC 11 displayed in the virtual conference space 8 is displayed only on the VR goggles 12 of user B.

[0134] Other participants in the conference also perform the same operations as the above-mentioned user B, and connect the notebook PCs and VR goggles of each participant to the virtual conference space 8.

[0135] <Operation of the VR operation controller> When user A or user B presses the menu display button 114 on the VR operation controller 6, 13, the Bluetooth communication control unit 120 of the VR operation controller 6, 13 transmits a menu display command to the VR goggles 5, 12 via Bluetooth.

[0136] The wireless LAN communication control unit 101 of the VR goggles 5, 12 transmits this command to the virtual conference service. When the virtual conference service receives this command, the display data generation unit 25 generates a menu list related to the virtual conference and displays it in the virtual conference space 8. An example of this menu list is shown in FIG.

[0137] 9 shows a menu list screen 200 displayed in the virtual conference space 8. The menu list screen 200 has a participant list button 201, a whiteboard generation button 202, and a settings button 203. The participant list button 201 is a button for displaying a list of participants currently attending a conference identified by a conference ID. The whiteboard generation button 202 is a button for creating a virtual electronic whiteboard in the virtual conference space 8. The settings button 203 is a button for displaying various setting screens such as language and character size.

[0138] Furthermore, when user A or user B presses the pointer display button 115 on the VR operation controller 6, 13, the Bluetooth communication control unit 120 of the VR operation controller 6, 13 transmits a pointer display command including information on the orientation and movement of the own device to the VR goggles 5, 12 via Bluetooth. The wireless LAN communication control unit 101 of the VR goggles 5, 12 transmits this command to the virtual conference service. When the virtual conference service receives this command, the pointer position calculation unit 27 calculates the display position of the pointer 170 based on the orientation and movement information of the VR operation controller 6, 13. The virtual display control unit 26 displays the pointer 170 at the calculated display position in the virtual conference space 8.

[0139] While the user is pressing the pointer display button 115, the Bluetooth communication control unit 120 of the VR operation controller 6, 13 periodically (for example, every 100 milliseconds) transmits a pointer display command including information on the orientation and movement of the own device to the VR goggles 5, 12. The wireless LAN communication control unit 101 of the VR goggles 5, 12 transmits this command to the virtual conference service. The pointer position calculation unit 27 calculates the display position of the pointer 170 based on the orientation and movement information of the VR operation controller 6, 13. When the pointer 170 moves and there is a change in the display of the virtual conference space 8, the virtual display control unit 26 of the virtual conference service transmits the display data of the virtual conference space 8 to the VR goggles 5, 12. Therefore, the pointer 170 is controlled to move in the virtual conference space 8.

[0140] When user A or user B presses the Confirm button 116 on the VR operation controller 6, 13, the Bluetooth communication control unit 120 of the VR operation controller 6, 13 transmits a Confirm command via Bluetooth to the VR goggles 5, 12. The wireless LAN communication control unit 101 of the VR goggles 5, 12 transmits this command to the virtual conference service. When the virtual conference service receives this command, it determines that the menu item corresponding to the display position of the pointer 170 has been selected (pressed), and performs an operation according to that menu.

[0141] When user A (or user B) moves the pointer 170 to the whiteboard generation button 202 shown in Fig. 9 and presses the confirm button 116, the VR operation controller 6, 13 transmits a confirm command to the virtual conference service via the VR goggles 5, 12. When the virtual conference service receives this command, the whiteboard generation and management unit 21 generates a virtual electronic whiteboard, and the virtual display control unit 26 displays it in the virtual conference space 8. Note that the screen when the virtual electronic whiteboard is generated is blank. That is, blank virtual electronic whiteboard display data is displayed in the virtual conference space 8.

[0142] <Generating virtual electronic whiteboard display data> On this virtual electronic whiteboard, users can display screen data of notebook PCs 4 and 11, and can display any display content (photos, figures, etc.). In Fig. 1, the screens of two virtual electronic whiteboards (virtual electronic whiteboard display data 9c, virtual electronic whiteboard display data 9d) are displayed. In the virtual electronic whiteboard display data 9c, screen data of notebook PC 14 operated by user C who is participating in the virtual conference is displayed.

[0143] The process and operation of displaying the screen data of the notebook PC 14 of user C on the virtual electronic whiteboard will be described. When user C starts the virtual conference app of the notebook PC 14 and performs the same operation as user B described above, the notebook PC 14 and the VR goggles 15 connect to the virtual conference space 8. When user C then selects the remote desktop selection button displayed on the notebook PC 14, the screen data of the notebook PC 14 is transmitted to the virtual conference service, and the screen data of the notebook PC 14 (virtual electronic whiteboard display data 9c) is displayed in the virtual conference space 8.

[0144] 10 shows the notebook PC display data 210 of the notebook PC 14 displayed in the virtual conference space 8. In the notebook PC display data 210, a movement bar 211 is added to a screen display area 212 in which the screen data of the notebook PC 14 in the virtual conference space 8 is displayed. The virtual electronic whiteboard display data 9c is displayed in the screen display area 212. When the user C presses the pointer display button 115 on the VR operation controller 16 and holds the pointer 170 over the movement bar 211, the screen display area 212 of the notebook PC 14 becomes movable. When the user C moves the pointer 170 while pressing the pointer display button 115, the screen display area 212 of the notebook PC 14 also moves accordingly.

[0145] When user C performs this operation to overlay screen display area 212 of notebook PC 14 on the virtual electronic whiteboard (blank virtual electronic whiteboard display data), virtual display control unit 26 of the virtual conference service copies the screen data of notebook PC 14 onto the screen of the virtual electronic whiteboard, and the screen data of notebook PC 14 (virtual electronic whiteboard display data 9c) is displayed on virtual electronic whiteboard 40. In addition, virtual display control unit 26 deletes screen display area 212 and movement bar 211 on which the screen data of notebook PC 14 was displayed.

[0146] <Action or Processing> 11A and 11B show a sequence diagram of operations subsequent to the operation in which user A moves the pointer 170 to the virtual electronic whiteboard display data 9c in order to display the screen of the virtual electronic whiteboard display data 9c on the electronic whiteboard 2. FIG.

[0147] S41: User A presses the pointer display button 115 on the VR operation controller 6.

[0148] S42: The Bluetooth communication control unit 120 of the VR operation controller 6 transmits a pointer display command including information on the orientation and movement of its own device to the VR goggles 5 via Bluetooth.

[0149] S43: The wireless LAN communication control unit 101 of the VR goggles 5 transmits this command to the virtual conference service.

[0150] S44: When the communication control unit 23 of the virtual conference service receives the command, the pointer position calculation unit 27 calculates the display position of the pointer 170 based on the information on the position and movement of the VR operation controller 6. The virtual display control unit 26 determines the icon and position of the pointer 170 in the virtual conference space 8 (pointer display processing).

[0151] S45: The virtual display control unit 26 of the virtual conference service transmits the updated display data of the virtual conference space 8, including the pointer 170, to the VR goggles 5 via the communication control unit 23.

[0152] S46: The wireless LAN communication control unit 101 of the VR goggles 5 receives the display data, and the display control unit 100 displays the updated display data.

[0153] S47 to S58: While the user is pressing the pointer display button 115, the Bluetooth communication control unit 120 of the VR operation controller 6 periodically (for example, every 100 milliseconds) transmits a pointer display command including information on the orientation and movement of the own device to the VR goggles 5. The wireless LAN communication control unit 101 of the VR goggles 5 transmits this command to the virtual conference service. The pointer position calculation unit 27 of the virtual conference service calculates the display position of the pointer 170 based on the information on the orientation and movement of the VR operation controller 6. When the pointer 170 moves and there is a change in the display of the virtual conference space 8, the virtual display control unit 26 of the virtual conference service transmits display data of the virtual conference space 8 to the VR goggles 5 via the communication control unit 23. The wireless LAN communication control unit 101 of the VR goggles 5 receives the display data, and the display control unit 100 displays the updated display data.

[0154] S59: Next, the user A moves the pointer 170 to the display area of ​​the virtual electronic whiteboard display data 9c while pressing the pointer display button 115. Then, the user A presses and holds the pointer display button 115 on the VR operation controller 6 for a predetermined period of time while the pointer 170 is overlapped with the virtual electronic whiteboard display data 9c.

[0155] S60: The Bluetooth communication control unit 120 of the VR operation controller 6 transmits a pointer display command including information on the orientation and movement of its own device to the VR goggles 5 via Bluetooth at predetermined time intervals.

[0156] S61: The wireless LAN communication control unit 101 of the VR goggles 5 transmits this command to the virtual conference service. Here, it is assumed that there is no change in the orientation and movement of the VR operation controller 6.

[0157] S62: When the communication control unit 23 of the virtual conference service receives this command, the pointer position calculation unit 27 calculates the display position of the pointer 170 based on the information on the orientation and movement of the VR operation controller 6. Here, it is assumed that there is no change in the orientation and movement of the VR operation controller 6.

[0158] S63 to S65: While the user A holds down the pointer display button 115 on the VR operation controller 6 for a predetermined period of time with the pointer 170 over the virtual electronic whiteboard display data 9c, steps S63 to S65 are repeatedly executed.

[0159] S66: When the amount of change in the display position of the pointer 170 continues to be equal to or less than a predetermined value a predetermined number of times (i.e., a predetermined time), the synchronization control unit 22 detects that the pointer display button 115 has been pressed and held while the pointer 170 is in the same position (an example of operation information), and accepts selection of the virtual electronic whiteboard display data 9c at the display position of the pointer 170. Since the user may have simply placed the pointer 170 over the virtual electronic whiteboard display data 9c for a predetermined time without intending to synchronize, the synchronization control unit 22 asks the user whether it is OK to synchronize.

[0160] The synchronization control unit 22 passes information to the virtual display control unit 26 that allows the user to select an action corresponding to the display position of the pointer 170. Upon receiving this information, the virtual display control unit 26 generates display data for a message saying "Do you want to display this screen on the electronic whiteboard?" and displays it in the virtual conference space 8.

[0161] S67: The virtual display control unit 26 of the virtual conference service transmits the updated display data of the virtual conference space 8, including the display of the above message, to the VR goggles 5 via the communication control unit 23.

[0162] S68: The wireless LAN communication control unit 101 of the VR goggles 5 receives the display data, and the display control unit 100 displays the updated display data.

[0163] 12 shows an inquiry screen 230 displayed in the virtual conference space 8 and viewable by the VR goggles 5. The inquiry screen 230 displays a message 231, a YES button 232, and a NO button 233. Here, a case where user A presses the YES button 232 will be described.

[0164] S69 to 80: User A moves the pointer 170 to the display area of ​​the YES button 232 while pressing the pointer display button 115. While the user presses the pointer display button 115, the Bluetooth communication control unit 120 of the VR operation controller 6 periodically (for example, every 100 milliseconds) transmits a pointer display command including information on the orientation and movement of the own device to the VR goggles 5. The wireless LAN communication control unit 101 of the VR goggles 5 transmits this command to the virtual conference service. The pointer position calculation unit 27 of the virtual conference service calculates the display position of the pointer 170 based on the information on the orientation and movement of the VR operation controller 6. When the pointer 170 moves and there is a change in the display of the virtual conference space 8, the virtual display control unit 26 of the virtual conference service transmits the display data of the virtual conference space 8 to the VR goggles 5 via the communication control unit 23. The wireless LAN communication control unit 101 of the VR goggles 5 receives the display data, and the display control unit 100 displays the updated display data. As a result, the pointer 170 of the virtual conference space 8 moves to the YES button 232.

[0165] S81: User A presses the confirmation button 116 with the pointer 170 overlapping the YES button 232.

[0166] S82: The Bluetooth communication control unit 120 of the VR operation controller 6 transmits a confirmation command to the VR goggles 5 via Bluetooth.

[0167] S83: The wireless LAN communication control unit 101 of the VR goggles 5 transmits this command to the virtual conference service.

[0168] S84: When the communication control unit 23 of the virtual conference service receives this command, the synchronization control unit 22 detects the pressing of the YES button 232 based on the position of the pointer 170, and determines that the virtual electronic whiteboard display data 9c specified by the long press of the pointer display button 115 in step S66 has been selected. The synchronization control unit 22 transmits the virtual electronic whiteboard display data 9c to the electronic whiteboard 2 in the conference room via the communication control unit 23.

[0169] S85: The LAN communication control unit 72 of the electronic whiteboard 2 receives the virtual electronic whiteboard display data 9c, and the display control unit 71 displays the virtual electronic whiteboard display data 9c as screen data on the display 480. Note that since the virtual electronic whiteboard display data 9c was originally a copy of the PC display data of the notebook PC 14 operated by user C, when the screen data of the notebook PC 14 is updated, the virtual electronic whiteboard display data 9c in the virtual conference space 8 is updated in synchronization with this, and further the virtual electronic whiteboard display data 9c displayed by the electronic whiteboard 2 is also updated.

[0170] S86: User A removes the VR goggles 5 and writes on the electronic whiteboard 2 in order to write characters or the like on the display screen.

[0171] S87: The contact position detection unit 70 of the electronic whiteboard 2 detects the contact position of the pen tip of the electronic pen 490, and the drawing data generation unit 74 generates drawing data. The display control unit 71 displays the handwritten drawing data on the display 480.

[0172] S88: The LAN communication control unit 72 of the electronic whiteboard 2 transmits the handwritten drawing data to the virtual conference service in real time.

[0173] S89: The communication control unit 23 of the virtual conference service receives the drawing data, and the synchronization control unit 22 passes the drawing data to the virtual display control unit 26. The virtual display control unit 26 updates the virtual electronic whiteboard display data 9c by merging the handwritten drawing data with the screen data of the virtual electronic whiteboard display data 9c.

[0174] S90-1: The communication control unit 23 of the virtual conference service transmits the updated virtual electronic whiteboard display data 9c to the VR goggles 5.

[0175] S90-2: The wireless LAN communication control unit 101 of the VR goggles 5 receives the virtual electronic whiteboard display data 9c, and the display control unit 100 updates the virtual electronic whiteboard display data 9c.

[0176] Fig. 13 shows a display example of the electronic whiteboard 2 on which drawing data is displayed and the virtual conference space 8. In Fig. 13, the electronic whiteboard 2 displays screen data of the notebook PC 14 of user C. Handwritten drawing data 150 is displayed on the screen data of the electronic whiteboard 2. The same drawing data 150 is also displayed in the virtual conference space 8 in the virtual electronic whiteboard display data 9c.

[0177] In this way, when a user selects any one of a plurality of virtual electronic whiteboards in the virtual conference space 8 with the pointer 170 or the like, the screen data of the virtual electronic whiteboard can be synchronized with the real electronic whiteboard 2. Furthermore, the drawing data on the real electronic whiteboard 2 can be reflected in real time in the display data of the virtual electronic whiteboard selected by the user.

[0178] In this embodiment, the notebook PC display data 210 of the notebook PC 14 is displayed as the virtual electronic whiteboard display data 9c, and the virtual electronic whiteboard display data 9c is reflected (synchronized) in the screen data of the electronic whiteboard 2. Conversely, the virtual conference service may display the screen data of the electronic whiteboard 2 (virtual electronic whiteboard display data) in the virtual conference space 8. In this case, it is sufficient that the electronic whiteboard 2 has a remote desktop control unit. If the remote desktop control unit transmits the screen data of the electronic whiteboard 2 to the virtual conference service, the virtual conference service can generate a virtual electronic whiteboard and display the screen data of the electronic whiteboard 2 as virtual electronic whiteboard display data in the virtual conference space 8 immediately after generation.

[0179] The acceptance of the selection of the virtual electronic whiteboard display data 9c may be confirmed by pressing and holding the pointer display button 115 overlapping with the virtual electronic whiteboard display data 9c. Furthermore, in this embodiment, a case has been described in which two virtual electronic whiteboards are displayed in the virtual conference space 8, but the number of virtual electronic whiteboards may be one or three or more.

[0180] Furthermore, in this embodiment, a method for selecting screen data of a virtual electronic whiteboard to be displayed on the electronic whiteboard 2 from among a plurality of virtual electronic whiteboards displayed in the virtual conference space 8 has been described in which the pointer display button 115 on the VR operation controller 6 is pressed and held down with the pointer 170 over the virtual electronic whiteboard. However, when the user overlays the pointer 170 on the virtual electronic whiteboard, the frame of the virtual electronic whiteboard is highlighted, and the virtual electronic whiteboard display data may be selected by pressing the confirm button 116 in this state.

[0181] 14 shows virtual electronic whiteboard display data 9c in which frame 160 is highlighted by the user hovering pointer 170 over the virtual electronic whiteboard. The user presses confirmation button 116 with frame 160 displayed. This operation information is transmitted from VR operation controller 6 to VR goggles 5, and from VR goggles 5 to the virtual conference service, and synchronization control unit 22 accepts the selection of virtual electronic whiteboard display data 9c. The user can select virtual electronic whiteboard display data without having to press and hold pointer display button 115.

[0182] <Major Effects> The virtual space system 300 of this embodiment enables handwriting on any one of a plurality of virtual electronic whiteboards displayed in the virtual conference space 8 using the real electronic whiteboard 2, improving user convenience. The user can easily select any one of the virtual electronic whiteboards in the virtual conference space 8 as the electronic whiteboard 2 on which to write using the real electronic whiteboard 2 by using the VR operation controller 6, improving user convenience.

[0183] [Second embodiment] When multiple virtual electronic whiteboards are generated in the virtual conference space 8 and there is actually one electronic whiteboard 2, the single electronic whiteboard 2 is shared by the multiple virtual electronic whiteboards. However, if a user synchronizes first virtual electronic whiteboard display data (an example of first display data) with the electronic whiteboard 2 and another user synchronizes second virtual electronic whiteboard display data (an example of second display data) with the electronic whiteboard 2, there is a risk that the screen of the electronic whiteboard 2 will suddenly switch to the second virtual electronic whiteboard display data while the user is handwriting on the first virtual electronic whiteboard display data on the electronic whiteboard 2.

[0184] In this embodiment, a virtual space system 300 that, when another user synchronizes a second virtual electronic whiteboard with the electronic whiteboard 2, invalidates the synchronization operation under a predetermined condition will be described.

[0185] In this embodiment, the functional block diagram of FIG. 6 described in the above embodiment can be used.

[0186] <Disable synchronization between virtual and electronic whiteboard> The electronic whiteboard 2 is in a "handwriting input state" from a state in which the electronic pen 490 is in contact with the display 480 (pen-down state) until a predetermined time has elapsed after the pen is released from the touch panel (pen-up state). The handwriting input state indicates that the user is currently writing by hand on the electronic whiteboard 2.

[0187] User C is also present in the conference room 17, and participates in the conference using the notebook PC 14, VR goggles 15, and VR operation controller 16, just like user A. As described in the first embodiment, the screen data of the electronic whiteboard 2 and the virtual electronic whiteboard display data 9c are synchronized, and while user A is inputting handwriting into the electronic whiteboard 2 (in the handwriting input state), user C wants to write handwriting on the screen of the virtual electronic whiteboard display data 9d. User C presses the pointer display button 115 on the VR operation controller 16 to display the pointer 170, and presses and holds the pointer display button 115 with the pointer 170 superimposed on the virtual electronic whiteboard display data 9d. The synchronization control unit 22 determines that the electronic whiteboard 2 is in the handwriting input state, and displays a message in the virtual conference space 8 via the virtual display control unit 26, stating, "This screen cannot be displayed on the electronic whiteboard because it is being used by another user."

[0188] 15 shows an example of a message 241 displayed in the virtual conference space 8. This message 241 is displayed, for example, superimposed on the virtual electronic whiteboard display data 9d. User C checks the message 241 and can determine that the virtual electronic whiteboard display data 9d cannot be synchronized with the electronic whiteboard 2.

[0189] The display of the message 241 when the frame 160 is displayed on the virtual electronic whiteboard display data as shown in Fig. 14 will be described. User C presses the pointer confirmation button 116 with the pointer 170 over the virtual electronic whiteboard display data 9d. The synchronization control unit 22 determines that the electronic whiteboard 2 is in a handwriting input state, and causes the message 241 to be displayed in the virtual conference space 8 via the virtual display control unit 26.

[0190] After that, user A finished handwriting input on the electronic whiteboard 2 and the handwriting input state was released. In other words, a predetermined time has passed since the pen was lifted. Since the electronic whiteboard 2 transmits handwritten drawing data to the virtual conference service in real time, the virtual conference service can detect the pen-up and the predetermined time that has passed since the pen-up.

[0191] After a predetermined time has elapsed since the pen was lifted, user C presses and holds the pointer display button 115 with the pointer 170 over the virtual electronic whiteboard display data 9d. The synchronization control unit 22 determines that the electronic whiteboard 2 is not in a state for handwriting input. Because the virtual electronic whiteboard display data 9c was already synchronized with the electronic whiteboard 2, the synchronization control unit 22 causes the virtual conference space 8 to display a confirmation message, "Do you want to display this screen on the electronic whiteboard?" via the virtual display control unit 26.

[0192] 16 is an example of a synchronization confirmation screen 250 displayed in the virtual conference space 8. On the synchronization confirmation screen 250, a YES button 252 and a NO button 253 are displayed together with a message 251. When the YES button 252 is pressed, the synchronization control unit 22 of the virtual conference service transmits display data of the virtual electronic whiteboard display data 9d to the electronic whiteboard 2 in the conference room via the communication control unit 23. The electronic whiteboard 2 displays the received screen data.

[0193] 14, the same is true when a frame 160 is displayed on the virtual electronic whiteboard display data, and user C presses the pointer confirmation button 116 with the pointer 170 over the virtual electronic whiteboard display data 9d. The synchronization control unit 22 determines that the electronic whiteboard 2 is not in a state for handwriting input. Since the virtual electronic whiteboard display data 9c has already been synchronized with the electronic whiteboard 2, the synchronization control unit 22 causes a message 251 to be displayed in the virtual conference space 8 via the virtual display control unit 26.

[0194] <Major Effects> When there are multiple participants in a conference room, while one participant is displaying handwritten drawings on the electronic whiteboard 2 on a virtual electronic whiteboard displayed in the virtual conference space 8, it is possible to prevent the screen of another virtual electronic whiteboard displayed in the virtual conference space 8 from being displayed on the electronic whiteboard 2. This makes it possible to prevent the display from switching to an unintended display while the user is handwriting, improving user convenience.

[0195] [Third embodiment] In the first embodiment, the user specifies the virtual electronic whiteboard in the virtual conference space 8 to be displayed on the electronic whiteboard 2 by pressing and holding the pointer display button 115 on the VR operation controller 6. In this embodiment, when there are multiple virtual electronic whiteboards in the virtual conference space 8, the user displays a list of thumbnail images of the virtual electronic whiteboard display data on the electronic whiteboard 2, and specifies the virtual electronic whiteboard display data to be synchronized with the electronic whiteboard 2 on the electronic whiteboard 2.

[0196] <About the function> Fig. 17 shows a functional configuration diagram of the virtual conference server 1 of this embodiment. Fig. 17 shows only the functions of the virtual conference server 1. In Fig. 17, components with the same reference numerals as those in Fig. 6 perform similar functions, so there are cases where only the main components of this embodiment are mainly described. Fig. 17 also includes a thumbnail generation unit 130. The thumbnail generation unit 130 generates a thumbnail image of the virtual electronic whiteboard display data displayed in the virtual conference space 8.

[0197] <Action or Processing> 18 and 19, a process will be described in which the interactive whiteboard 2 displays a list of thumbnail images of the virtual electronic whiteboard display data, and the user designates the virtual electronic whiteboard display data to be synchronized with the interactive whiteboard 2.

[0198] Fig. 18 is a sequence diagram explaining the process in which the electronic whiteboard 2 displays a thumbnail image of the virtual electronic whiteboard display data being displayed in the virtual conference space 8. Fig. 19 shows an example of a virtual whiteboard list display button 260 possessed by the electronic whiteboard 2. The virtual whiteboard list display button 260 may be a button (soft key) that is part of a menu list, or may be a hard key. When multiple virtual electronic whiteboards are generated in the virtual conference space 8, the virtual conference service may notify the electronic whiteboard 2 of this, and the electronic whiteboard 2 may automatically display the virtual whiteboard list display button 260.

[0199] In this way, even if there is only one virtual electronic whiteboard in the virtual conference space 8, the virtual whiteboard list display button 260 is displayed, and the user can reduce the effort required to press the button. In other words, when there is only one virtual electronic whiteboard in the virtual conference space 8, the synchronization control unit 22 of the virtual conference service may automatically synchronize the virtual electronic whiteboard display data with the electronic whiteboard 2, even if the user does not select the virtual electronic whiteboard display data as in the first embodiment and this embodiment.

[0200] S91: The virtual conference application executed on the electronic whiteboard 2 has a virtual whiteboard list display button 260 (see FIG. 19) for displaying a list of thumbnail images of the virtual electronic whiteboard display data in the virtual conference space 8. First, user A selects this virtual whiteboard list display button 260.

[0201] S92: In response to the pressing of the virtual whiteboard list display button 260, the LAN communication control unit 72 of the interactive whiteboard 2 transmits a virtual whiteboard list acquisition command to the virtual conference service.

[0202] S93: When the communication control unit 23 of the virtual conference service receives this command, the thumbnail generation unit 130 generates thumbnail images of the virtual electronic whiteboard display data being displayed in the virtual conference space 8. When there are two virtual electronic whiteboards as in Fig. 1, the thumbnail generation unit 130 generates a thumbnail image of the virtual electronic whiteboard display data 9c and a thumbnail image of the virtual electronic whiteboard display data 9d.

[0203] S94: The communication control unit 23 of the virtual conference service transmits to the electronic whiteboard 2 these thumbnail images and the identification information of the virtual electronic whiteboard corresponding to each thumbnail image in a response message to the virtual whiteboard list acquisition command.

[0204] S95: The LAN communication control unit 72 of the interactive whiteboard 2 receives these data, and the display control unit 71 displays the received thumbnail images on the display 480. An example of how the thumbnail images are displayed on the interactive whiteboard 2 is shown in Fig. 20(a).

[0205] Fig. 20(a) shows thumbnail images 261, 262 of the virtual electronic whiteboard display data displayed by the electronic whiteboard 2. As shown in Fig. 20, when the virtual whiteboard list display button 260 is pressed, thumbnail images 261, 262 of the virtual electronic whiteboard display data present in the virtual conference space 8 are displayed in a list. The user selects thumbnail images 261, 262 with the hand H or electronic pen 490. Identification information of the selected thumbnail images 261, 262 is sent to the virtual conference service.

[0206] The electronic whiteboard 2 only needs to display a selectable virtual electronic whiteboard to be synchronized, and may display a list of identification information of the virtual electronic whiteboards in addition to the thumbnail images of the virtual electronic whiteboard display data.

[0207] FIG. 20(b) shows the user name list 46 of the virtual electronic whiteboard displayed by the electronic whiteboard 2. The user selects a user name with his / her hand H or electronic pen 490. The selected user name is sent to the virtual conference service. The user name is the owner of the notebook PC copied onto the virtual electronic whiteboard, and is the identification information of the virtual electronic whiteboard. The identification information of the virtual electronic whiteboard displayed as the user name list 46 may be, for example, the creation number (e.g., a sequential number) of the virtual electronic whiteboard generated in the virtual conference space 8, the user's location, a facial photo, an avatar, etc.

[0208] S96: The user double-clicks, for example, the thumbnail image of the virtual electronic whiteboard display data 9c.

[0209] S97: The LAN communication control unit 72 of the electronic whiteboard 2 transmits the identification information of the virtual electronic whiteboard associated with the thumbnail image of the virtual electronic whiteboard display data 9c to the virtual conference service, including the identification information in the virtual whiteboard screen acquisition command.

[0210] S98: When the communication control unit 23 of the virtual conference service receives this command, the synchronization control unit 22 accepts the selection of the virtual electronic whiteboard display data specified by the identification information of the virtual electronic whiteboard. The synchronization control unit 22 acquires the selected virtual electronic whiteboard display data 9c from the virtual conference space 8. The synchronization control unit 22 transmits the virtual electronic whiteboard display data 9c to the electronic whiteboard 2 via the communication control unit 23.

[0211] S99: The LAN communication control unit 72 of the electronic whiteboard 2 receives the virtual electronic whiteboard display data 9c, and the display control unit 71 causes the display 480 to display the virtual electronic whiteboard display data 9c as screen data.

[0212] S100: Next, the user writes by hand on the electronic whiteboard 2 displaying the virtual electronic whiteboard display data 9c.

[0213] S101: The contact position detection unit 70 of the electronic whiteboard 2 detects the contact position of the pen tip of the electronic pen 490, and the drawing data generation unit 74 generates drawing data. The display control unit 71 displays the handwritten drawing data by superimposing it on the screen of the display 480.

[0214] S102: The LAN communication control unit 72 of the electronic whiteboard 2 transmits the drawing data to the virtual conference service in real time.

[0215] S103: When the communication control unit 23 of the virtual conference service receives the drawing data, the virtual display control unit 26 synthesizes (superimposes) the drawing data on the virtual electronic whiteboard display data 9c, thereby updating the virtual electronic whiteboard display data 9c. A display example of the electronic whiteboard 2 and the virtual conference space 8 with synchronized drawing data is similar to that shown in FIG.

[0216] <Screen data switching after synchronization> After the multiple virtual electronic whiteboard display data are synchronized with the electronic whiteboard 2, the electronic whiteboard 2 may slide the screen data of the display 480 by one screen, and switch between displaying the multiple virtual electronic whiteboard display data 9c, 9d displayed in the virtual conference space 8, one screen at a time.

[0217] 21 is a diagram illustrating an operation in which a user switches the virtual electronic whiteboard display data to be displayed on the display 480. The virtual electronic whiteboard display data 9c, 9d of the virtual conference space 8 has already been selected, and the screen data 41, 42 of the electronic whiteboard 2 and the virtual electronic whiteboard display data 9c, 9d have been synchronized. The electronic whiteboard 2 stores information about which virtual electronic whiteboard display data 9c, 9d the screen data 41, 42 are associated with.

[0218] For example, when the user flicks left while virtual electronic whiteboard display data 9c is being displayed, the electronic whiteboard 2 displays virtual electronic whiteboard display data 9d as screen data 42. When the user flicks right while virtual electronic whiteboard display data 9d is being displayed, the electronic whiteboard 2 displays virtual electronic whiteboard display data 9c as screen data 41. When virtual electronic whiteboard display data that is not being displayed by the electronic whiteboard 2 is updated in the virtual conference space 8, the synchronization control unit 22 of the virtual conference service transmits the updated virtual electronic whiteboard display data to the electronic whiteboard 2 via the communication control unit 23. The virtual electronic whiteboard display data that is not being displayed by the electronic whiteboard 2 refers to all virtual electronic whiteboard display data that is not the virtual electronic whiteboard display data that the user last selected to be synchronized.

[0219] Therefore, once the virtual electronic whiteboard display data is synchronized with the electronic whiteboard 2, the user can easily switch between the screen data 41 and 42. Immediately after the user slides the screen, the screen data 41 and 42 can be displayed in a state synchronized with the virtual electronic whiteboard display data 9c and 9d.

[0220] <Major Effects> According to the present embodiment, the user can select multiple virtual electronic whiteboards in the virtual meeting space 8 to be displayed on the electronic whiteboard 2 by operating the electronic whiteboard 2, improving user convenience. Even if there is only one electronic whiteboard 2, multiple virtual electronic whiteboard display data can be saved in the virtual meeting space 8, so the user can switch between the virtual electronic whiteboard display data and write by hand.

[0221] <Other application examples> The above describes the best mode for carrying out the present invention using examples, but the present invention is not limited to these examples in any way, and various modifications and substitutions can be made within the scope that does not deviate from the gist of the present invention.

[0222] For example, in this embodiment, the user operates the VR operation controller to select the virtual electronic whiteboard display data, but voice operation may also be used.

[0223] In the present embodiment, screen data of an electronic whiteboard is displayed in the virtual conference space 8, but screen data of a projector, a HUD (Head Up Display) device, a television receiver, a digital camera, a medical device, a smartphone, a tablet terminal, a game console, a PDA (Personal Digital Assistant), a wearable PC, etc. may be displayed. Information that can be displayed in two or three dimensions can be displayed in the virtual conference space 8.

[0224] The electronic whiteboard may be called an electronic whiteboard, an electronic information board, or the like. The present embodiment is not limited to electronic whiteboards and can be suitably applied to any information processing device having a touch panel. Examples of information processing devices equipped with a touch panel include PCs, tablet terminals, and smartphones having a touch panel. These are usually general-purpose information processing devices, but when an application that functions as a display device is executed, the user can operate them as a display device.

[0225] In addition, in this embodiment, display data is displayed in the virtual conference space 8, but avatars of participants may be displayed in the virtual conference space 8. An avatar is an alter ego of a participant created by computer graphics. By wearing the VR goggles 5, 12, 15, participants can grasp the avatars of each participant in the virtual conference space 8 as if they were present in the real space.

[0226] While the participant is writing on the electronic whiteboard 2, the virtual display control unit 26 may display an avatar of the participant writing on the synchronized virtual electronic whiteboard display data. In this case, the electronic whiteboard 2 uses face recognition or login information to identify the participant writing, and transmits the participant's identification information to the virtual conference service. The virtual display control unit 26 displays the avatar of the participant identified by the identification information near the synchronized virtual electronic whiteboard display data. Furthermore, the virtual display control unit 26 may give the avatar a pen icon, and display the icon's pen tip following the tip of the drawing data displayed in real time on the virtual electronic whiteboard display data. The icon may be displayed semi-transparently so as not to hide the drawing data or the virtual electronic whiteboard display data.

[0227] Furthermore, when multiple pieces of virtual electronic whiteboard display data are displayed in the virtual conference space 8 and any one of them is synchronized with the real electronic whiteboard 2, the virtual display control unit 26 may emphasize the virtual electronic whiteboard display data that is synchronized with the electronic whiteboard 2. Emphasis means, for example, blinking the frame of the virtual electronic whiteboard display data or displaying it somewhat larger.

[0228] Moreover, the configuration example in Fig. 6 and the like is divided according to main functions in order to facilitate understanding of the processing by the virtual space system 300. The present invention is not limited by the manner in which the processing units are divided or the names of the processing units. The processing of the virtual space system 300 can be divided into even more processing units depending on the processing contents. Also, it can be divided so that one processing unit includes even more processes.

[0229] Each function of the above-described embodiments can be realized by one or more processing circuits. Here, the term "processing circuit" in this specification includes a processor programmed to execute each function by software, such as a processor implemented by an electronic circuit, and a device such as an ASIC (Application Specific Integrated Circuit), a DSP (digital signal processor), an FPGA (field programmable gate array), or a conventional circuit module designed to execute each function described above.

[0230] Embodiments of the present invention provide significant improvements in computer power and functionality. These improvements allow users to utilize computers that provide more efficient and robust interaction with tables, which are ways of storing and presenting information in information processing devices. Furthermore, embodiments of the present invention provide a better user experience through the use of more efficient, powerful and robust user interfaces. Such user interfaces provide better interaction between humans and machines.

[0231] <Additional Notes> [Appendix 1] A virtual space system for displaying display data displayed in a virtual space on a real electronic whiteboard, comprising: a synchronization control unit that synchronizes the display data selected by a user with screen data displayed by the electronic whiteboard when a plurality of pieces of display data are displayed in the virtual space; A virtual space system having the above configuration. [Appendix 2] the synchronization control unit accepts a selection of any one of the plurality of display data displayed in the virtual space, 2. The virtual space system according to claim 1, wherein the display data for which a selection has been accepted is synchronized with the screen data displayed by the electronic whiteboard. [Appendix 3] The virtual space system includes an information processing system, The information processing system communicates with a device that displays display data to be displayed in the virtual space; the information processing system has a virtual display control unit that displays a pointer indicating the display data in the virtual space based on a command transmitted from the device; A virtual space system as described in Appendix 1 or 2, wherein when predetermined operation information is received from the device while the pointer is overlapping with the display data in the virtual space, the synchronization control unit synchronizes the display data overlapping with the pointer with the screen data displayed by the electronic whiteboard. [Appendix 4] The virtual space system described in Appendix 3, wherein the specified operation information includes an operation of pressing and holding a button on a controller that communicates with the device. [Appendix 5] The virtual space system includes an information processing system, The information processing system communicates with a device that displays any display data in the virtual space; the information processing system has a virtual display control unit that displays a pointer indicating the display data in the virtual space based on a command transmitted from the device; When the pointer overlaps with the display data displayed in the virtual space, the virtual display control unit displays the display data in an emphasized manner; The virtual space system described in Appendix 1 or 2, wherein the synchronization control unit synchronizes the display data overlapping the pointer with the screen data displayed by the electronic whiteboard when predetermined operation information is received from the device while the display data is being displayed in an emphasized manner. [Appendix 6] When a plurality of pieces of display data are displayed in the virtual space and the synchronization control unit synchronizes a first piece of display data with the electronic whiteboard, A virtual space system as described in Appendix 3 or 4, wherein when the synchronization control unit receives specific operation information that synchronizes the second display data with the electronic whiteboard, the synchronization control unit disables the specific operation information that synchronizes the second display data with the electronic whiteboard. [Appendix 7] Even when the synchronization control unit synchronizes the first display data with the electronic whiteboard, The virtual space system described in Appendix 6, wherein when a predetermined time has elapsed since receiving a pen-up from the electronic whiteboard, the synchronization control unit receives specified operation information that synchronizes the second display data with the electronic whiteboard, and does not invalidate the operation information, but synchronizes the second display data with the screen data displayed by the electronic whiteboard. [Appendix 8] A list of the virtual electronic whiteboards generated in the virtual space is transmitted to the electronic whiteboard; When the identification information of the virtual electronic whiteboard selected on the electronic whiteboard is received, the synchronization control unit accepts selection of the display data identified by the identification information of the virtual electronic whiteboard; 2. The virtual space system according to claim 1, wherein the display data for which a selection has been accepted is synchronized with screen data displayed by the electronic whiteboard. [Appendix 9] Transmitting thumbnail images of one or more of the display data displayed in the virtual space to the electronic whiteboard; The virtual space system described in Appendix 8, wherein when identification information of the display data selected on the electronic whiteboard is received, the synchronization control unit synchronizes the display data corresponding to the identification information of the selected display data with the screen data displayed by the electronic whiteboard. [Appendix 10] After the synchronization control unit synchronizes the first display data with the electronic whiteboard, When the second display data is synchronized with the electronic whiteboard, The virtual space system described in Appendix 7, wherein the synchronization control unit synchronizes by sending the display data updated in the virtual space to the electronic whiteboard even if the second display data is display data that is not displayed by the electronic whiteboard. [Appendix 11] The virtual space system according to claim 10, wherein the electronic whiteboard switches from display data currently being displayed on the electronic whiteboard to display data not being displayed on the electronic whiteboard in response to a user's operation. [Appendix 12] When the synchronization control unit synchronizes the display data selected by the user with the screen data displayed by the electronic whiteboard, 11. The virtual space system according to any one of claims 1 to 10, wherein drawing data handwritten on the electronic whiteboard is displayed so as to be superimposed on the display data selected by a user. [Appendix 13] screen data received from a terminal device and to be displayed on a display of the terminal device is displayed in the virtual space; the display data is a copy of the screen data, The virtual space system according to any one of appendices 1 to 12, wherein the synchronization control unit synchronizes the selected display data with the screen data of the electronic whiteboard when the display data to which the screen data has been copied is selected by a user. [Explanation of symbols]

[0232] 1 Virtual Conference Server 2. Electronic whiteboard 4,11,14 Notebook PC 5,12,15 VR goggles 6,13,16 VR operation controller 8. Virtual Meeting Spaces 9 Display Data [Prior art documents] [Patent documents]

[0233] [Patent Document 1] Special Publication No. 2022-521117

Claims

1. A virtual space system for displaying display data displayed in a virtual space on a real electronic whiteboard, comprising: a synchronization control unit that synchronizes the display data selected by a user with screen data displayed by the electronic whiteboard when a plurality of pieces of display data are displayed in the virtual space; A virtual space system having the above configuration.

2. the synchronization control unit accepts a selection of any one of the plurality of display data displayed in the virtual space, 2. The virtual space system according to claim 1, wherein the display data for which a selection has been accepted is synchronized with the screen data displayed by the electronic whiteboard.

3. The virtual space system includes an information processing system, The information processing system communicates with a device that displays display data to be displayed in the virtual space; the information processing system has a virtual display control unit that displays a pointer indicating the display data in the virtual space based on a command transmitted from the device; The virtual space system of claim 1 or 2, wherein when the pointer is overlapped with the display data in the virtual space and a specified operation information is received from the device, the synchronization control unit synchronizes the display data overlapped with the pointer with the screen data displayed by the electronic whiteboard.

4. The virtual space system according to claim 3 , wherein the predetermined operation information includes an operation of pressing and holding a button on a controller that communicates with the device.

5. The virtual space system includes an information processing system, The information processing system communicates with a device that displays any display data in the virtual space; the information processing system has a virtual display control unit that displays a pointer indicating the display data in the virtual space based on a command transmitted from the device; When the pointer overlaps with the display data displayed in the virtual space, the virtual display control unit displays the display data in an emphasized manner; The virtual space system of claim 1 or 2, wherein the synchronization control unit synchronizes the display data overlapping the pointer with the screen data displayed by the electronic whiteboard when predetermined operation information is received from the device while the display data is being displayed in an emphasized manner.

6. When a plurality of pieces of display data are displayed in the virtual space and the synchronization control unit synchronizes a first piece of display data with the electronic whiteboard, The virtual space system of claim 3, wherein when the synchronization control unit receives specific operation information that synchronizes the second display data with the electronic whiteboard, the synchronization control unit disables the specific operation information that synchronizes the second display data with the electronic whiteboard.

7. Even when the synchronization control unit synchronizes the first display data with the electronic whiteboard, The virtual space system described in claim 6, wherein when a predetermined time has elapsed since receiving a pen-up from the electronic whiteboard, a predetermined operation information for synchronizing the second display data with the electronic whiteboard is received, the synchronization control unit does not invalidate the operation information, but synchronizes the second display data with the screen data displayed by the electronic whiteboard.

8. A list of the virtual electronic whiteboards generated in the virtual space is transmitted to the electronic whiteboard; When the identification information of the virtual electronic whiteboard selected on the electronic whiteboard is received, the synchronization control unit accepts selection of the display data identified by the identification information of the virtual electronic whiteboard; 2. The virtual space system according to claim 1, wherein the display data for which the selection has been accepted is synchronized with screen data displayed by the electronic whiteboard.

9. Transmitting thumbnail images of one or more of the display data displayed in the virtual space to the electronic whiteboard; The virtual space system of claim 8, wherein when identification information of the display data selected on the electronic whiteboard is received, the synchronization control unit synchronizes the display data corresponding to the identification information of the selected display data with the screen data displayed by the electronic whiteboard.

10. After the synchronization control unit synchronizes the first display data with the electronic whiteboard, When the second display data is synchronized with the electronic whiteboard, The virtual space system according to claim 7, wherein the synchronization control unit synchronizes the display data updated in the virtual space by sending the display data to the electronic whiteboard, even if the second display data is display data that is not displayed by the electronic whiteboard.

11. The virtual space system according to claim 10 , wherein the electronic whiteboard switches, in response to a user's operation, from display data currently being displayed on the electronic whiteboard to display data not being displayed on the electronic whiteboard.

12. When the synchronization control unit synchronizes the display data selected by the user with the screen data displayed by the electronic whiteboard, 2. The virtual space system according to claim 1, wherein drawing data handwritten on said electronic whiteboard is displayed so as to be superimposed on said display data selected by a user.

13. screen data received from a terminal device and to be displayed on a display of the terminal device is displayed in the virtual space; the display data is a copy of the screen data, The virtual space system according to claim 1 , wherein the synchronization control unit, when the display data to which the screen data is copied is selected by a user, synchronizes the selected display data with the screen data of the electronic whiteboard.

14. An information processing system for displaying display data displayed in a virtual space on a real electronic whiteboard, comprising: a synchronization control unit that synchronizes the display data selected by a user with screen data displayed by the electronic whiteboard when a plurality of pieces of display data are displayed in the virtual space; An information processing system having the above configuration.

15. A synchronization method performed by a virtual space system that displays display data displayed in a virtual space on a real electronic whiteboard, comprising the steps of: A synchronization method for synchronizing a piece of display data selected by a user with screen data displayed by the electronic whiteboard when a plurality of pieces of display data are displayed in the virtual space.

16. An electronic whiteboard that can communicate with a virtual space system via a network, the virtual space system displaying display data on a real electronic whiteboard, a LAN communication control unit that receives the display data in order to synchronize the display data selected by a user with the screen data displayed by the electronic whiteboard when a plurality of display data are displayed in the virtual space; a display control unit that displays the display data received by the LAN communication control unit on a display; An electronic whiteboard with a

Images