Display methods, information provision methods, display systems, information processing devices, head-mounted displays

Abstract

To prevent content from becoming smaller when content outside the display area is aggregated and displayed. [Solution] The present invention provides an information processing device 2 that can scroll a display area to display content input in a display extension area, which performs the following processes: extracting one or more content items from the content input in the display extension area based on the type of content, and aggregating the extracted one or more content items into the display area.

Description

Technical Field

[0001] The present invention relates to a display method, an information providing method, a display system, an information processing apparatus, and a head-mounted display.

Background Art

[0002] In an information processing apparatus such as an electronic blackboard that allows a user to input handwriting, there is known a technique of moving the content input into the display area outside the display area and making it possible to use an area wider than the display area (hereinafter referred to as a display expansion area).

[0003] A technique of collectively displaying search results of content on one electronic blackboard has been devised (see, for example, Patent Document 1). Patent Document 1 discloses a technique in which each electronic whiteboard to be searched is divided into a plurality of areas, after extracting a plurality of areas that match the search conditions, a screen in which the extracted areas are arranged in a state where they can be browsed in a list is displayed on the client terminal that is the search request source.

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in the conventional technique, since all the content outside the display area is aggregated and displayed, when the number of content is large, there is a problem that it is reduced and difficult to view.

[0005] In view of the above problems, the present invention provides a technique for suppressing each content from becoming small when aggregating and displaying content outside the display area.

Means for Solving the Problems

[0006] In view of the above problems, the present invention provides an information processing device that can scroll a display area to display content input in a display extension area, which performs the following processes: extracting one or more content items from the content input in the display extension area based on the type of content; and aggregating the extracted one or more content items in the display area. [Effects of the Invention]

[0007] This prevents content from becoming smaller when it is aggregated and displayed from outside the display area. [Brief explanation of the drawing]

[0008] [Figure 1] This diagram illustrates the general overview of the aggregated display. [Figure 2] This figure shows an example of the system configuration of a display system. [Figure 3] This is a hardware configuration diagram of an example of an interactive whiteboard. [Figure 4] This diagram shows the hardware configuration of an example tablet PC. [Figure 5] This is an example of a functional block diagram that explains the functions of an electronic whiteboard and a tablet PC in a display system by dividing them into blocks. [Figure 6] This diagram illustrates the correspondence between the display extension area and the display area. [Figure 7] This diagram illustrates how to copy the screen of a PC connected to an interactive whiteboard onto the whiteboard. [Figure 8] This diagram shows an example of what the electronic whiteboard screen looks like when a portion of the PC screen is copied onto it. [Figure 9] This diagram shows a display extension area where multiple content items are arranged. [Figure 10] This diagram schematically shows the content management table managed by the content management unit in the data storage unit. [Figure 11] This figure shows an example of a condition setting screen. [Figure 12]This is a diagram for explaining the content displayed in the display area when a graph is selected in the type specification column. [Figure 13] This is a diagram for explaining the arrangement rules using Contents A to F. [Figure 14] This is a diagram for explaining the arrangement rules using Contents A to F. [Figure 15] This is a diagram showing an example of Contents A to F after reduction. [Figure 16] This is a diagram showing an example of Contents A to F after reduction. [Figure 17] This is a diagram for explaining an example of the arrangement of content. [Figure 18] This is a diagram showing an example of the coordinates of the content before reduction. [Figure 19] This is a diagram showing an example of the content reduced at the reduction rate. [Figure 20] This is an example flowchart diagram for explaining the process of the electronic blackboard aggregating and displaying content. [Figure 21] This is a system configuration diagram of an example of a virtual conference system. [Figure 22] This is a hardware configuration diagram of an example of a VR goggle. [Figure 23] This is a hardware configuration diagram of an example of a VR operation controller. [Figure 24] This is a hardware configuration diagram of an example of a virtual conference server. [Figure 25] This is an example functional block diagram for explaining the functions of the virtual conference server and the electronic blackboard by dividing them into blocks. [Figure 26] This is an example of a functional block diagram for explaining the functions of the virtual conference server, PC, VR goggle, and VR operation controller by dividing them into blocks. [Figure 27] This is a diagram showing an example display of the virtual conference space displayed on the VR goggle. [Figure 28] This is a sequence diagram for explaining the drawing process of drawing data by the VR operation controller. [Figure 29]It is a sequence diagram for explaining the drawing process of drawing data on an electronic blackboard. [Figure 30] It is a diagram showing an example of the display of an electronic blackboard and an example of the display of a virtual conference space. [Figure 31] It is a diagram showing an example of a virtual conference space in which the content has been moved to the left. [Figure 32] It is a diagram showing an example of the content management table managed by the content management department. [Figure 33] It is a diagram showing an example of the virtual PC screen data of a virtual conference space. [Figure 34] It is a diagram showing the content of the virtual whiteboard displayed on the electronic blackboard. [Figure 35] It is a diagram showing an example of the content and virtual PC screen data existing in a virtual conference space. [Figure 36] It is a diagram showing an example of a virtual conference space in which the content that was on the virtual whiteboard has been moved upward. [Figure 37] It is a diagram showing an example of the content management table managed by the content management department in the virtual conference space of FIG. 36. [Figure 38] It is a diagram showing an example of the content handwritten on the electronic blackboard. [Figure 39] It is a diagram showing an example of a virtual whiteboard synchronized with the drawing data handwritten on the electronic blackboard. [Figure 40] It is a diagram showing an example of the content management table in the virtual conference space of FIG. 39. [Figure 41] It is a diagram showing an example of a virtual whiteboard in which content 215 is arranged on the upper side and content 211 is arranged on the lower side. [Figure 42] It is a diagram showing an example of the virtual whiteboard after reduction and an example of the content displayed on the display by the electronic blackboard. [Figure 43] It is a sequence diagram for explaining the processing or operation performed by the virtual conference system. [Figure 44]This figure shows an example of the system configuration of the display system according to the third embodiment. [Figure 45] This is a functional block diagram illustrating an example of how the functions of the conference server are divided into blocks according to the third embodiment. [Figure 46] This is a functional block diagram illustrating an example of the functions of an electronic whiteboard divided into blocks according to the third embodiment. [Figure 47] This is a functional block diagram illustrating an example of how the functions of a tablet PC are divided into blocks according to the third embodiment. [Figure 48] This is an example sequence diagram illustrating the process of an interactive whiteboard and two tablet PCs initiating a remote meeting via a conference server. [Figure 49] This diagram illustrates an example of what can be displayed on an electronic whiteboard screen. [Figure 50] This diagram illustrates an example of what can be displayed on an electronic whiteboard screen. [Figure 51] This is an example sequence diagram illustrating the synchronization process between an electronic whiteboard and a tablet PC. [Figure 52] This figure shows an example of content placed in the display extension area. [Figure 53] This figure shows an example of content placed in the display extension area. [Figure 54] This is a sequence diagram illustrating an example of the process of placing multiple content items into a whiteboard area. [Figure 55] This is an example flowchart illustrating the actions taken by the conference server after it receives bar graph content from the electronic whiteboard. [Figure 56] This is an example sequence diagram illustrating the process of sending the edit flag. [Figure 57] This diagram illustrates an example of what can be displayed on an electronic whiteboard screen. [Figure 58] This diagram illustrates content 206 displayed on the electronic whiteboard's screen. [Figure 59] This diagram illustrates the thumbnail display of a reduced whiteboard area on a tablet PC. [Figure 60] This is an example sequence diagram illustrating the process when the edit flag is cleared. [Figure 61] This figure shows an example of content placement within the display extension area. [Figure 62] This diagram shows an example of content that an interactive whiteboard can display on its whiteboard area. [Figure 63] This diagram shows an example of a settings screen displayed on an interactive whiteboard. [Figure 64] This is an example sequence diagram illustrating the process of sending content to an interactive whiteboard and a tablet PC. [Figure 65] This diagram shows an example of content that an interactive whiteboard can display on its whiteboard area. [Figure 66] A functional configuration diagram of an example of a conference server in the fifth embodiment is shown. [Figure 67] This figure shows an example of instruction sentences learned through instruction tuning and their corresponding outputs (tasks). [Figure 68] This is an example flowchart illustrating the process of an electronic whiteboard that accepts voice commands for aggregated display. [Modes for carrying out the invention]

[0009] Hereinafter, an example of an embodiment for carrying out the present invention will be described with reference to the drawings, including an electronic whiteboard and a display method performed by the electronic whiteboard.

[0010] <Outline of the aggregated display in this embodiment> Figure 1 is a diagram illustrating the schematic of the aggregated display in this embodiment. The electronic whiteboard in this embodiment aggregates and displays only the types of content that the user wants to see on the screen. As shown in Figure 1(a), multiple content items 201 to 205 exist in the display extension area 252. These may be handwritten or imported from a connected PC (Personal Computer).

[0011] When aggregating the content of the display extension area 252, the user can set the type of content they want to aggregate. For example, suppose the user selects "Graph (Image)" as the type. The interactive whiteboard automatically determines the type of each content. As shown in Figure 1(b), the interactive whiteboard extracts content 201 and 205, which are determined to be graphs, and places content 201 and 205 in order from the top left of the display area 251. The interactive whiteboard will also shrink content 201 and 205 if they do not fit in the display area 251.

[0012] Thus, in this embodiment, the electronic whiteboard displays only the types of content the user wants to see in the display area 251. Therefore, even if there is a lot of content in the display extension area 252, the size of each piece of content can be suppressed. Consequently, the screen becomes easier to view compared to conventional designs.

[0013] <About Terminology> The device can be anything that displays drawing data. In this embodiment, the device is, for example, an electronic whiteboard, a tablet PC, a notebook PC, a smartphone, a projector, etc., but is not limited to these.

[0014] Content refers to data that is entered into or displayed on the display area of ​​an electronic whiteboard. Content can include, for example, handwritten drawing data, text data (fonts), shapes, screen data, images, etc., as long as it is displayable data. Drawing data consists of one or more handwritten lines. Drawing data may also be referred to as handwritten data.

[0015] An electronic whiteboard is a device that uses a touch panel to display handwritten drawings and shapes (including finished drawings) in real time on a screen or similar display. Electronic whiteboards may also have features such as network connectivity and the ability to convert handwritten characters into font data using OCR.

[0016] The display area is the area on which a device displays content on a single screen. The extended display area is an area that extends beyond the display area and is larger than the display area.

[0017] [First Embodiment] <Example System Configuration> Referring to Figure 2, the system configuration of the display system 300 will be explained. Figure 2 is a diagram showing an example of the system configuration of the display system 300. An electronic whiteboard 2A is located at site X, an electronic whiteboard 2B is located at site A, and a tablet PC 3 is located at site B. In the display system 300, the devices at each site are connected to each other via a communication network N, with the electronic whiteboard 2A enabling communication. The devices are, for example, the electronic whiteboard 2B and the tablet PC 3, but these are just examples.

[0018] The communication network N is, for example, the internet, but it could also be an on-premise network such as a company's internal network. The number of devices and locations are just examples; an electronic whiteboard 2A may exist as a standalone unit. There may be two or more devices, and multiple devices may be deployed at a single location.

[0019] Electronic whiteboard 2A (an example of the first information processing device) is an example of an information processing device that aggregates and displays content in this embodiment. Electronic whiteboard 2B also has similar functions. Any electronic whiteboard among electronic whiteboard 2A and electronic whiteboard 2B is referred to as "electronic whiteboard 2". Electronic whiteboard 2A manages meeting information, etc., and manages the start and end of meetings, participant logins, etc. The service provided by the meeting application executed on electronic whiteboard 2A is referred to as the meeting service.

[0020] The electronic whiteboard 2B (an example of a second information processing device) and tablet PC 3 are examples of information processing devices operated by meeting participants (users). The information processing device is not limited to the electronic whiteboard 2B or tablet PC 3; any information processing device with communication capabilities and capable of running a meeting application, such as a PC, smartphone, or PDA (Personal Digital Assistant), is acceptable.

[0021] The electronic whiteboard 2 and tablet PC 3 preferably have touch panels. Users can write by hand using an electronic pen or their fingertips. Even if the electronic whiteboard 2 does not have a touch panel, users can write by hand using a pointing device such as a mouse. In addition to handwritten drawing data, the electronic whiteboard 2 and tablet PC 3 can also display content such as text converted from drawing data, voice-input text, figures, and PC screens, and this content can also be shared between locations.

[0022] The interactive whiteboard 2B and tablet PC 3 can be connected to the interactive whiteboard 2A. This means the display system 300 can be used for online meetings. Content entered on interactive whiteboards 2A, 2B, and tablet PC 3 is combined onto a single screen, allowing for sharing across all locations. The IP address and password for interactive whiteboard 2A are either pre-configured on interactive whiteboard 2B and tablet PC 3, or have been pre-distributed to participants via email or other means.

[0023] When interactive whiteboards 2A and 2B and tablet PC 3 participate in an online meeting, interactive whiteboard 2A transmits content and audio data entered into it to interactive whiteboard 2B and tablet PC 3. Similarly, content and audio data entered into interactive whiteboard 2B are transmitted to interactive whiteboard 2A, which then transmits them to tablet PC 3. Tablet PC 3 displays the received content and outputs audio data. Similarly, content and audio data entered into tablet PC 3 are transmitted to interactive whiteboard 2A, which then transmits them to interactive whiteboard 2B. Interactive whiteboard 2B displays the received content and outputs audio data. The online meeting is conducted by repeating this process between interactive whiteboard 2B and tablet PC 3.

[0024] <Example Hardware Configuration> <<Electronic Whiteboard>> Figure 3 is a hardware configuration diagram of an example of the electronic whiteboard 2. As shown in Figure 3, the electronic whiteboard 2 is equipped with a CPU (Central Processing Unit) 401, ROM (Read Only Memory) 402, RAM (Random Access Memory) 403, SSD (Solid State Drive) 404, network I / F 405, and external device connection I / F (Interface) 406.

[0025] Of these, the CPU 401 controls the overall operation of the electronic whiteboard 2. The ROM 402 stores programs used to start the OS, such as the IPL (Initial Program Loader). The RAM 403 is used as the work area for the CPU 401. The SSD 404 stores various data, such as programs for the electronic whiteboard 2. The network I / F 405 controls communication with the communication network N. The external device connection I / F 406 is an interface for connecting various external devices. In this case, external devices include, for example, a USB (Universal Serial Bus) memory 430 and external devices (microphone 440, speaker 450, camera 460).

[0026] Furthermore, the electronic whiteboard 2 is equipped with a capture device 411, a GPU 412, a display controller 413, a contact sensor 414, a sensor controller 415, an electronic pen controller 416, a short-range communication circuit 419, and an antenna 419a for the short-range communication circuit 419, a power switch 422, and selector switches 423.

[0027] Of these, the capture device 411 displays the display information of the PC 470 connected via an HDMI (registered trademark, High-Definition Multimedia Interface) cable as a still image or video. The GPU (Graphics Processing Unit) 412 is a semiconductor chip specializing in graphics. The display controller 413 controls and manages the screen display in order to output the output image from the GPU 412 to a display means such as the display 480. The contact sensor 414 detects when an electronic pen 490 or the user's hand H comes into contact with the display 480. The sensor controller 415 controls the processing of the contact sensor 414. The contact sensor 414 detects the input coordinates using an infrared blocking method. This input coordinate detection is performed using two light-emitting and receiving devices installed on both upper ends of the display 480. The light-emitting element (laser) in the light-emitting and receiving device rotates and scans an infrared beam parallel to the display 480 within a 90-degree range. The infrared beam is reflected by reflective members provided around the display 480. The light-receiving element receives light that returns along the same optical path as the light it emitted. The light-receiving and light-emitting device, which is a contact sensor 414, outputs the position where infrared light is blocked by an object (position on the light-receiving element) to the sensor controller 415, and the sensor controller 415 identifies the coordinate position of the object's contact from the infrared blocking position information on these two light-receiving elements. The electronic pen controller 416 communicates with the electronic pen 490 via Bluetooth (registered trademark; hereinafter omitted) to determine whether the touch on the display 480 is from the pen tip or the pen tail. The electronic pen 490 has a switch to switch between a mode for operation with the pen tip and a mode for operation with the pen tail, and transmits the set mode information via Bluetooth. When the electronic whiteboard 2 is in pen tip mode, it draws a line at the position of the coordinate sequence that has been touched and traced, and when it is in pen tail mode, it functions as an eraser and erases the line drawn at the position of the coordinate sequence that has been touched and traced. The near-field communication circuit 419 is a communication circuit such as NFC (Near Field Communication) or Bluetooth.The power switch 422 is a switch for turning the power of the electronic whiteboard 2 on and off. The selection switches 423 are a group of switches for adjusting, for example, the brightness and color of the display 480.

[0028] Furthermore, the electronic whiteboard 2 is equipped with a bus line 410. The bus line 410 is an address bus, data bus, etc., for electrically connecting each component, such as the CPU 401 shown in Figure 3.

[0029] Furthermore, the contact sensor 414 is not limited to an infrared blocking type; it may also be a capacitive type touch panel that identifies the contact position by detecting a change in capacitance. The contact sensor 414 may also be a resistive type touch panel that identifies the contact position by voltage changes between two opposing resistive films. The contact sensor 414 may also be an electromagnetic induction type touch panel that identifies the contact position by detecting electromagnetic induction caused by contact between an object and the display. In addition, the contact sensor 414 may use various detection means. Moreover, the electronic pen controller 416 may determine whether there is a touch not only on the tip and end of the electronic pen 490, but also on the part of the electronic pen 490 that the user holds and other parts of the electronic pen.

[0030] <<Tablet PC>> Figure 4 shows the hardware configuration of an example of a tablet PC 3. As shown in Figure 4, the tablet PC 3 is built by a computer 520 and includes a CPU 521, ROM 522, RAM 523, HD (Hard Disk) 524, HDD (Hard Disk Drive) controller 525, display 526, external device connection I / F (Interface) 527, bus line 528, network I / F 529, pointing device 530, and media I / F 531. The pointing device 530 is a touch panel.

[0031] Of these, the CPU 521 controls the overall operation of the tablet PC 3. The ROM 522 stores programs used to drive the CPU 521, such as the IPL. The RAM 523 is used as the work area for the CPU 521. The HD 524 stores various data such as programs. The HDD controller 525 controls the reading or writing of various data to the HD 524 according to the control of the CPU 521. The display 526 displays various information such as cursors, menus, windows, characters, or images. The external device connection I / F 527 is an interface for connecting various external devices. In this case, external devices include, for example, the camera 540, microphone 541, speaker 542, USB (Universal Serial Bus) memory, and printer. The network I / F 529 is an interface for data communication using a network. The bus line 528 is an address bus, data bus, etc., for electrically connecting each component such as the CPU 521 shown in Figure 4. The pointing device 530 is a type of input means used for selecting and executing various instructions, selecting processing targets, and moving the cursor. The media interface 531 controls the reading or writing (storage) of data to or from the recording medium 532, such as flash memory.

[0032] <About the features> Next, with reference to Figure 5, the functional configuration of the display system 300 will be described. Figure 5 is an example of a functional block diagram that explains the functions of the electronic whiteboard 2A, electronic whiteboard 2B, and tablet PC 3 in the display system 300 by dividing them into blocks.

[0033] <<Electronic Whiteboard 2A>> The electronic whiteboard 2A includes a content receiving unit 21, a display control unit 22, a type determination unit 23, a contact position detection unit 24, a data storage unit 25, an operation reception unit 26, a content transmission unit 27, a drawing data generation unit 28, a content management unit 29, and an aggregation processing unit 30. Each of these parts of the electronic whiteboard 2A is a function or means of functioning, realized by any of the components shown in Figure 3 operating according to instructions from the CPU 401 in accordance with a program deployed from the SSD 404 to the RAM 403.

[0034] The content receiving unit 21 is a means for receiving handwritten drawing data, etc., on the display area 251 from other devices. Specifically, the content receiving unit 21 receives content such as drawing data and figures transmitted from the electronic whiteboard 2B or the tablet PC 3.

[0035] The content transmission unit 27 transmits content entered at its own location to equipment at other locations. The content displayed at its own location includes content directly entered into the electronic whiteboard 2A and content transmitted from the electronic whiteboard 2B or tablet PC 3 to the electronic whiteboard 2A.

[0036] The operation reception unit 26 receives operations on the electronic whiteboard 2A. For example, it determines the location of a button or the like at the coordinates where the electronic pen 490 or hand H has made contact, and accepts the button press.

[0037] The display control unit 22 displays content such as drawing data and shapes input to the device, displays meeting information, and displays content received from the electronic whiteboard 2B or tablet PC 3. The display control unit 22 scrolls the display area 251 to move content to the display extension area 252, which is larger than the display area. In addition, the display control unit 22 displays the content from the display extension area 252 in the display area 251 in conjunction with the scrolling operation.

[0038] The type determination unit 23 determines the type of each piece of content. The type determination unit 23 has a type determination model that has been machine-trained using training data for determining the type of content. The type determination model extracts features of the content using a feature extraction model suitable for images, such as a CNN (Convolutional Neural Network), and determines the type of content.

[0039] The contact position detection unit 24 detects the contact position of the user's hand H or the electronic pen 490. The drawing data generation unit 28 generates drawing data from the sequence of coordinate points input from the contact position detection unit 24 when the user writes by hand, and also generates shapes such as circles and rectangles.

[0040] The content management unit 29 manages content using the data storage unit 25. As will be described in detail later, the data storage unit 25 stores a content management table that manages each piece of content, including content transmitted from other devices. When new content is input, the content management unit 29 registers that information in the content management table, and updates the information in the content management table when content is updated.

[0041] The aggregation processing unit 30 performs processing related to content aggregation. The aggregation processing unit 30 places the type of content specified by the user within the display area 251 and also shrinks it so that it fits within the display area 251. The aggregation processing unit 30 can also search for content that matches keywords.

[0042] <<Functions of the 2B Electronic Whiteboard>> The functions of the interactive whiteboard 2B are the same as those of the interactive whiteboard 2A, so we will omit the explanation.

[0043] <<Tablet PC>> The tablet PC 3 includes a content transmission unit 31, a content reception unit 32, a data storage unit 33, a display control unit 34, and a contact position detection unit 35. Each of these parts of the tablet PC 3 is a function or means of functioning that is realized by any of the components shown in Figure 4 operating according to instructions from the CPU 521 that follow a program deployed from the HD 524 to the RAM 523. The data storage unit 33 is built on the HD 524, RAM 523, etc.

[0044] The content transmission unit 31 transmits content such as drawing data and shapes input to the device to the electronic whiteboard 2A.

[0045] The content receiving unit 32 receives content such as drawing data and shapes transmitted from the electronic whiteboard 2A. Therefore, the electronic whiteboard 2A and the tablet PC 3 share content in real time.

[0046] The display control unit 34 displays content such as drawing data and shapes input by the machine, displays meeting information, displays content received from the electronic whiteboard 2A, and updates the display area 251 in response to scrolling operations (displaying a portion of the display extension area 252).

[0047] The contact position detection unit 35 detects the contact position of the user's hand H or the electronic pen 490. The data storage unit 33 stores information about each piece of content, including content transmitted from other devices.

[0048] The tablet PC 3 may also have functions similar to those of the electronic whiteboard 2A, such as a drawing data generation unit.

[0049] <About the display extension area and the display area> Referring to Figure 6, the display extension area 252 and the display area 251 will be explained. Figure 6 is a diagram illustrating the correspondence between the display extension area 252 and the display area 251. The display area 251 is the area displayed on the display 480. The display extension area 252 includes the display area 251 and is the area where content that cannot be displayed in the display area 251 is placed. The size of the display extension area 252 depends on the image memory, etc., but a larger area is secured than that of the display area 251.

[0050] The display area 251 shows a menu bar 254 and a consolidated display menu button 253. The consolidated display menu button 253 is used to display the condition setting screen 260 (see Figure 11), which will be described later.

[0051] The coordinates of the display area 251 are defined using a coordinate system where the origin is the pixel position at the top-left corner of the display 480 of the electronic whiteboard 2A, with the right direction being the positive X-axis and the downward direction being the positive Y-axis. That is, Coordinates of display area 251 = (Number of pixels in the X-axis direction from the pixel at the origin, Number of pixels in the Y-axis direction from the pixel at the origin) That is the case.

[0052] The coordinates of the display extension area 252 are defined as a coordinate system where, assuming the display area 251 of the electronic whiteboard 2A's display 480 is fixed in its initial position, the pixel position at the top-left corner of the display area is the origin, with the right direction being the positive X-axis, the left direction being the negative X-axis, the down direction being the positive Y-axis, and the up direction being the negative Y-axis.

[0053] <Copy screen data from the connected device to the display area of ​​the interactive whiteboard.> Based on Figure 7, a method for copying the entire screen or a portion of the screen of a PC connected to the electronic whiteboard 2A to the electronic whiteboard 2A when the screen is displayed in the display area 251 will be explained.

[0054] When the electronic whiteboards 2A and 2B are powered on, they display a whiteboard on the display 480. The menu bar 254 displayed at the bottom of the whiteboard includes, in addition to the aggregate display menu button 253, a connection menu button that displays a screen for connecting to other electronic whiteboards, a full-screen capture button that copies the entire screen data as a still image to the whiteboard when the display information of the external PC 470 is displayed on the display 480 as a still image or video, a partial screen capture button 218 that copies a part of the screen data as a still image to the whiteboard, and a whiteboard display button that switches from the display of the external PC 470's display to the whiteboard display. When the electronic whiteboard 2A and the PC 470 are connected with an HDMI® cable, the capture device 411 of the electronic whiteboard 2A displays the screen data of the PC 470 on the display 480. An example of this display is shown in Figure 7.

[0055] Here, the user touches the partial screen capture button 218 on the menu bar 254 to copy the bar graph 217 displayed on the display 480 to the whiteboard. After touching the upper left corner of the bar graph 217, the user moves their finger to the lower right corner of the bar graph 217 while keeping it in contact with the display 480, and then lifts their finger from the display 480. The electronic whiteboard 2A then displays a rectangle with the line connecting the start and end points of the touch as its diagonal for a predetermined time. Subsequently, when the user touches the whiteboard display button on the menu bar 254, the display on the display 480 switches from the PC 470 screen to the whiteboard display. An example of the display on the electronic whiteboard 2A's display 480 at this time is shown in Figure 8. In this way, the bar graph 217 displayed on the PC 470 screen can be copied to the whiteboard on the electronic whiteboard 2A.

[0056] <Example Content> Figure 9 shows a display extension area 252 with multiple content items arranged. In Figure 9, five content items are displayed. Content 201 is a bar graph entered on the electronic whiteboard 2A at site X. Content 202 is drawing data entered into electronic whiteboard 2B at location A. Content 203 is drawing data entered into electronic whiteboard 2B at location A. Content 204 is text data entered into tablet PC 3 at site B. Content 205 is a pie chart entered into tablet PC 3 at site B.

[0057] Content entered into a device at one location is shared with the display extension area 252 of the electronic whiteboard 2A. When a user enters content into a device at any location, they move the display area 251 to find an empty space and enter the content. The user can move the display area 251 by swiping to display content other than the currently displayed area 251 in the display area 251. In this way, the display extension area 252 has the content arrangement shown in Figure 9.

[0058] Note that content 201 shown in Figure 9 was copied from the screen data of the connected PC, and content 205 was received from tablet PC 3 at site B. The content displayed in the display extension area 252 includes drawing data, shapes, text data, image data stored in the electronic whiteboard, as well as image data acquired by the web browser and screen data acquired from the PC.

[0059] When adding content to the display extension area 252, the type determination unit 23 determines the type of content. That is, the type determination model maintains parameters that improve the accuracy of content classification by running a machine learning program using training data that can be classified into one of the following categories: "drawing (handwritten)", "shape", "text", "photograph", or "graph (image)". Note that the determination itself may be performed by an external server. This type determination model can be a CNN or similar, but existing models can be used as appropriate. For example, there is a model that combines a CNN with ResNet (Residual Neural Networks) to suppress vanishing gradients. ResNet is a model that enables learning in deep networks by branching the output from a certain layer of the neural network using residual connections, skipping one or more layers, and connecting and adding the outputs of the skipped layers.

[0060] When content is added to the display extension area 252, the type determination unit 23 determines whether it is "drawing (handwritten)", "shape", "text", "photo", or "graph (image)".

[0061] Figure 10 schematically shows the content management table managed by the content management unit 29 in the data storage unit 25. The electronic whiteboard 2B and tablet PC 3 also have similar information to that shown in Figure 10. The content management table has the following items: content ID, type, content, content addition location, content start coordinates, content width (an example of size information or second size information), and content height (an example of size information or second size information). • A content ID is an identifier that uniquely identifies a piece of content. The type is the type of content determined by the type determination unit 23. The content is registered when the data is text data or when the drawing data is converted to text data using character recognition, and it is a string representing the text data. • The content addition location is the location where the content was first entered. The content addition location can be the user's identification information. The starting coordinates of the content are those of the top-left corner of the bounding rectangle when the content is enclosed within that rectangle (coordinates in the display extension area 252). The coordinates of the top-left, top-right, bottom-right, and bottom-left corners of the content are just examples of positional information. • The content width is the horizontal length of the bounding rectangle when the content is enclosed within it. The content width is the difference between the top-left and top-right corners, or the difference between the bottom-right and bottom-left corners. • The content height is the vertical length of the bounding rectangle when the content is enclosed within it. The content height is the difference between the top-left and bottom-left corners, or the difference between the top-right and bottom-right corners.

[0062] <Specify the types of content to aggregate> As a method for displaying aggregated content, the display control unit 22 displays the condition setting screen 260 when the user presses the aggregated display menu button 253 shown in Figure 6.

[0063] Figure 11 shows an example of the condition setting screen 260. The condition setting screen 260 has a keyword field 261, a type specification field 262, a consolidation button 263, and a cancel button 264. The keyword field 261 is a field for setting a string to search for text data or drawing data. When the user enters a string in the keyword field 261 and presses the consolidation button 263, the consolidation processing unit 30 consolidates the content containing this string into the display area 251, and the display control unit 22 displays it.

[0064] The type specification field 262 is used to specify the type of content to be aggregated. The type specification field 262 displays a list of content types determined by the type determination unit 23 in a pull-down format. Therefore, the type specification field 262 can display all types of content in the display extension area 252 (only the types of content in the display extension area 252) without any omissions or excesses. When the user selects one type from the displayed types and presses the aggregate button 263, the aggregate processing unit 30 aggregates the content corresponding to the specified type into the display area 251, and the display control unit 22 displays it. Note that the user may select multiple types.

[0065] Additionally, it would be helpful to display the number of items available for each content type in the dropdown menu. Users can use this as a reference when selecting content types. For example, if there are only a few items, they might choose just two, or if there are many, they might narrow down their choices using keywords.

[0066] The keyword field 261 and the type specification field 262 may function as either an AND condition or an OR condition. In other words, when the user presses the aggregate button 263, the aggregate processing unit 30 aggregates content that matches the conditions into the display area 251 based on information about the content of the data storage unit 25.

[0067] Alternatively, the user may select one or more pieces of content, and the aggregation processing unit 30 may aggregate them in the display area 251. In this case, the list of content may be displayed as thumbnails.

[0068] <Placement rules for aggregated content> Next, we will explain the layout rules for the aggregated content, referring to Figures 12 to 15. Figure 12 is a diagram illustrating the content displayed in the display area 251 when a graph is selected in the type specification field 262. First, Figure 12 shows the coordinates of each graph and the display area 251. The coordinates of the top-left, top-right, bottom-right, and bottom-left corners of content 201 are (x3, y3), (x4, y3), (x4, y4), and (x3, y4). The coordinates of the top-left, top-right, bottom-right, and bottom-left corners of content 205 are (x5, y5), (x6, y5), (x6, y6), and (x5, y6). The coordinates of the top-left, top-right, bottom-right, and bottom-left corners of the display area 251 are (x1,y1), (x2,y1), (x2,y2), and (x1,y2).

[0069] Note that the upper left corner, upper right corner, lower right corner, and lower left corner are the edges of the content or display area.

[0070] Figure 13 illustrates the layout rules using contents A to F. As shown in Figure 13, the aggregation processing unit 30 divides the display extension area 252 into a left area and a right area by a perpendicular line 255 passing through the center point of the display area 251. (1) If content A to F exists only in the left area or only in the right area, they will be placed in the display area 251 in order from top to bottom. (2) If (1) is not satisfied, the number of content items in the left area and the number of content items in the right area are compared to determine the smaller number. The aggregation processing unit 30 then places this smaller number of content items in the left area and this smaller number of content items in the right area alternately in the display area 251 from top to bottom. In other words, the aggregation processing unit 30 places the above number of content items alternately in the display area 251 from top to bottom, such as the top of the left half, the top of the right half, the second from the top of the left half, and the second from the top of the right half. At this time, the content items in the left area are placed on the left side of the display area 251, and the content items in the right area are placed on the right side of the display area 251.

[0071] Furthermore, when adding content to the left or right half of the display area 251, if there is empty space to the right of the already placed content and the next content can be placed there, it will be placed to the right of the already placed content, rather than below it. (3) After the placement in (2), the aggregation processing unit 30 adds the content that exists only in the left area or only in the right area to the display area 251, starting from the top. At this time, the aggregation processing unit 30 places the content alternately in the left half and the right half of the display area 251, prioritizing the one with the smaller coordinate in the Y-axis direction.

[0072] Furthermore, when adding content to the left or right half of the display area 251, if there is empty space to the right of the already placed content and the next content can be placed there, it will be placed to the right of the already placed content, rather than below it.

[0073] In Figure 13, before aggregation, content A through D are located in the left-hand area, and content E and F are located in the right-hand area. Comparing the number of content items in the left-hand area with the number of content items in the right-hand area, the smaller number is 2. a. Therefore, in order of increasing coordinate value in the Y-axis direction, content B is first placed at the top of the left half of the display area 251. b. Next, content E is placed at the top of the right half of the display area 251. c. Next, content A is placed second from the top in the left half of display area 251. d. Next, content F is placed second from the top in the right half of the display area 251. e. Of the remaining contents C and D, content C is placed third from the top on the left half. f. Content D cannot be placed to the right of Content C. In this case, Content D is added to the left or right area where the last piece of content is located, whichever is higher. Therefore, the remaining Content D is placed fourth from the top in the left half.

[0074] Based on the above, contents A to F are arranged as shown in Figure 14. In Figure 14, the contents are displayed clustered towards the center of the display area. Note that the arrangement rules are just examples; the contents may also be arranged in the order of their coordinates from the top left to the bottom right of the display area 251, or in order of the largest or smallest area from the top left to the bottom right of the display area 251. Alternatively, contents A to F may be arranged using AI to minimize the amount of space used.

[0075] The problem of arranging blocks to minimize their area is generally known as the "packing problem." This is a combinatorial optimization problem aimed at arranging items of different sizes and shapes as efficiently as possible within a storage space. This problem is considered difficult to solve, and finding the optimal solution is especially challenging when the blocks come in various sizes and shapes. However, the following algorithms and methods are generally known.

[0076] 1. Greedy Algorithms The simplest method involves placing the blocks one by one in order. For example, there is the "maximum anticipation method," which places the blocks in order from largest to smallest, and the "minimum anticipation method," which places them in order from smallest to largest.

[0077] 2. Heuristic Algorithms - For example, there are methods such as "First Fit Decreasing (FFD)" and "Best Fit Decreasing (BFD)." These sort blocks by size and arrange them to efficiently fill the space.

[0078] 3. Backtracking This method involves partially exploring the search space for solutions to find a solution. It doesn't try all possibilities, but rather performs a certain degree of pruning during the search.

[0079] 4. Dynamic Programming This method efficiently finds solutions by dividing a problem into subproblems, solving each subproblem, and reusing the results. It is particularly applied to packing in 2D and 3D.

[0080] 5. Metaheuristic Algorithms Methods include genetic algorithms, simulated annealing, and particle swarm optimization. These are algorithms used to find approximate solutions and are particularly effective for large-scale problems.

[0081] 6. Integer Linear Programming (ILP) This method involves formulating the problem as an integer linear programming problem and finding the optimal solution. While commercial software can provide highly accurate solutions, it can be computationally expensive. By applying the packing problem, the electronic whiteboard 2 can be arranged so that a large proportion of the display area is occupied by content of a specified type or content related to a specified keyword, and can be displayed on the device. Furthermore, if multiple types of content are aggregated, the aggregation processing unit 30 may arrange the content so that similar types of content are grouped together. In this case, the aggregation processing unit 30 arranges only the same type of content first, in the same manner as described above, from top to bottom in the display area 251.

[0082] In the state shown in Figure 14, not all of the content A to F fit within the display area 251. Therefore, the aggregation processing unit 30 reduces the size so that all of the content A to F fit within the display area 251. Figure 15 shows the content A to F after reduction.

[0083] Since there is empty space to the right of content E and content F, you may move content E and F to the right to make them easier to view, as shown in Figure 16. In Figure 16, the content is displayed aligned to both sides of the display area.

[0084] The same placement rules apply to content 201 and 205 shown in Figure 12. Figure 17 illustrates the placement of contents 201 and 205. Since content 201 is located in the left area and content 205 is located in the right area, content 201 is displayed in the upper left of the display area 251, and content 205 is placed to the right of content 201. Using the coordinates in Figure 12, the aggregation processing unit 30 moves the top-left coordinates of content 201, (x3, y3), to coincide with the top-left coordinates of the display area 251, (0, 0), and moves the top-left coordinates of content 205, (x5, y5), to ((x4 - x3) + a, 0). The aggregation processing unit 30 places contents 201 and 205 with a distance a between them. This distance a can be, for example, 20, and serves as a gap to ensure readability.

[0085] In the state shown in Figure 17, content 205 extends beyond the X-axis of the display area 251, making it impossible to display the entire content 205. Therefore, the aggregation processing unit 30 reduces the overall size of content 201 and 205 so that content 205 also fits within the display area 251. If the user selects multiple types of content, the aggregation processing unit 30 may not reduce the overall size uniformly, but may vary the reduction ratio for each type. For example, the aggregation processing unit 30 may set the reduction ratio of graphs (less than 1) to be smaller than that of drawing data (mainly text). This makes it easier to fit the content within the display area 251 while ensuring readability. The user may also be able to set the reduction ratio for each type.

[0086] Furthermore, the aggregation processing unit 30 may detect the character size of the drawing data using an existing method, determine the content reduction ratio so that the character size does not fall below a certain level (i.e., to a certain size), and then determine the reduction ratio of other content so that other content fits within the display area with this reduction ratio.

[0087] Figure 18 shows the coordinates of content 201 and 205 before reduction. The coordinates of the bounding rectangle for the entire content 201 and 205 before reduction are as follows: The coordinates of the top-left corner are (0, 0), the top-right corner are (x4-x3+a+x6-x5, 0), the bottom-right corner is (x4-3+a+x6-x5, y4-y3), and the bottom-left corner is (0, y4-y3).

[0088] Here, the length (number of pixels) of the display area 251 in the X-axis direction is x2, and the length (number of pixels) from the left edge of content 201 to the right edge of content 205 is (x4-x3)+(x6-x5)+a. Therefore, the scaling ratio when shrinking content 201 and 205 to fit within the display area 251 is as follows. Reduction rate=(x2) / ((x4-x3)+(x6-x5)+a) Figure 19 shows content 201 and 205 scaled down to this reduction ratio. Content 201 and 205 fit within the display area 251, and the entire content is displayed.

[0089] When displaying multiple content items together, they are arranged according to the content placement rules described above. However, the total width (length in the X-axis direction) of the multiple content items may fit within the display area 251, while the height (length in the Y-axis direction) may extend beyond the display area 251. In this case, the aggregation processing unit 30 calculates a reduction ratio in the Y-axis direction and reduces the content accordingly. Furthermore, there may be cases where both the total width and height of the multiple content items extend beyond the display area 251. In this case, the reduction ratios in the X-axis direction and the Y-axis direction are calculated, and the content is reduced using the smaller reduction ratio.

[0090] <Action or process> Figure 20 is a flowchart illustrating the process by which the electronic whiteboard 2A aggregates and displays content. The process in Figure 20 can be executed, for example, after the electronic whiteboard 2A is started, but the aggregated display menu button 253 may only be displayed if content exists in the display extension area 252.

[0091] The operation reception unit 26 determines whether the aggregate display menu button 253 has been pressed (S11). If the aggregate display menu button 253 has been pressed (Yes in S11), it further determines whether the aggregate button 263 was pressed while a content type has been selected in the type specification field 262 or while a string has been entered in the keyword field 261 (S12).

[0092] If the determination in step S12 is Yes, the aggregation processing unit 30 refers to the data storage unit 25, extracts content that matches the conditions, and aggregates the content according to the arrangement rules (S13).

[0093] Next, the aggregation processing unit 30 determines whether the aggregated content extends beyond the display area 251 (S14). That is, it determines whether the width of the aggregated content is longer than the width of the display area 251 (X-axis direction), or whether the height of the aggregated content is longer than the height of the display area 251 (Y-axis direction).

[0094] If the aggregated content extends beyond the display area 251 (Yes in S14), the aggregation processing unit 30 determines whether the extension is in only one direction, either the X-axis or Y-axis (S15).

[0095] If only one direction is involved (Yes in S15), the aggregation processing unit 30 calculates the reduction ratio for the direction of overhang (S16).

[0096] If the reduction is not unidirectional (No. in S15), the aggregation processing unit 30 calculates the reduction ratio in the X-axis direction and the Y-axis direction and sets the smaller value as the reduction ratio (S17).

[0097] The aggregation processing unit 30 reduces all aggregated content by a predetermined reduction ratio, and the display control unit 22 displays it in the display area 251 (S18).

[0098] If the aggregated content does not extend beyond the display area 251 (No. in S14), the display control unit 22 displays the aggregated content as is (S19).

[0099] Note that on electronic whiteboard 2B, a unique display area is shown, so aggregated display is not performed. On electronic whiteboard 2B, the process shown in Figure 20 is executed when the aggregate button 263 is pressed on electronic whiteboard 2B. However, if electronic whiteboards 2A and 2B are set to synchronize, electronic whiteboard 2B may also aggregate and arrange one or more content items.

[0100] <Main effects> In this embodiment, the electronic whiteboard 2A displays only the types of content the user wants to see in the display area 251. Therefore, even if there is a lot of content in the display extension area 252, the size of each piece of content can be suppressed. Consequently, the screen is easier to view compared to conventional displays.

[0101] [Second Embodiment] This embodiment describes the aggregated display of content in a virtual meeting using a virtual meeting server.

[0102] Figure 21 shows the system configuration diagram of the virtual meeting system 700. This describes a scenario where a user at location 1 (conference room) participates in the virtual meeting using the electronic whiteboard 2, and users at locations 2 and 3 (homes) participate in the virtual meeting using VR (Virtual Reality) goggles and a VR control controller. Users at locations 2 and 3 (homes) display content displayed on their PCs in the virtual meeting, while users at location 1 (conference room) display content displayed on the electronic whiteboard 2 in the virtual meeting. Therefore, when the virtual meeting server aggregates and arranges the content, not only the content entered into the electronic whiteboard 2, but also the content displayed by the electronic whiteboard 2 and the content displayed on the PC can be aggregated and arranged in the display area, improving the readability of the content.

[0103] In a virtual meeting, the virtual meeting server 5 aggregates and displays content from each location. The virtual meeting server 5 transmits content placed in a virtual space (referred to as the virtual meeting space 18) to the electronic whiteboard 2 and VR goggles 10 and 12, and also places content received from the electronic whiteboard 2 and PCs 17 and 19 (an example of terminal devices) into the virtual space. The virtual meeting space 18 is formed in the memory of the virtual meeting server 5, and the content transmitted from the electronic whiteboard 2 and PCs 17 and 19 is written to the virtual meeting space 18.

[0104] When electronic whiteboard 2, PCs 17 and 19 request aggregated content display, the virtual meeting server 5 aggregates the content from each location into an area corresponding to display area 251 and transmits it to electronic whiteboard 2, PCs 17 and 19. The area corresponding to display area 251 is an area of ​​the same size as the display area 251 on electronic whiteboard 2. This will be explained in detail below.

[0105] <Example System Configuration> As shown in Figure 21, an electronic whiteboard 2 is placed in the conference room (location 1) where users A to C are located, user D is at home (location 2), and user E is at home (location 3). The virtual conference server 5, electronic whiteboard 2, PCs 17 and 19, and VR goggles 10 and 12 are connected to a communication network N such as the internet.

[0106] Virtual meeting server 5 is an example of an information processing device that aggregates and displays content (an example of a first information processing device). Virtual meeting server 5 runs a virtual meeting application. Hereinafter, the service provided by the virtual meeting application executed on virtual meeting server 5 will be referred to as the virtual meeting service. The virtual meeting space 18 is a 3D space, for example, 360°, that simulates a meeting room in a virtual space, but it may also be a planar space larger than the display area of ​​each device. Each device refers to the electronic whiteboard 2, PCs 17 and 19, and VR goggles 10 and 12. The data for the virtual meeting space 18 consists of screen data from each device. In the data for the virtual meeting space 18, the screen data of electronic whiteboard 2 (an example of a second information processing device) is called virtual whiteboard 18a. In the data for the virtual meeting space 18, the screen data of user D's PC 17 is called virtual PC screen data 18b. In the data for the virtual meeting space 18, the screen data of user E's PC 19 is called virtual PC screen data 18c. Virtual whiteboard 18a is synchronized with the content of electronic whiteboard 2, virtual PC screen data 18b is synchronized with the screen data of PC 17, and virtual PC screen data 18c is synchronized with the screen data of PC 19. VR goggles 10 and 12 can display the data for the virtual meeting space 18.

[0107] An electronic whiteboard 2 is installed in the conference room where users A, B, and C are present. Electronic whiteboard 2 is connected to the communication network N and can communicate with the virtual conference server 5.

[0108] User D is at home (or a satellite office, etc.), and a Wi-Fi router 8 is installed at home. User D is using a PC 17, VR goggles 10, and a VR control controller 11. The Wi-Fi router 8 at User D's home can connect to communication network N. User D's PC 17 and VR goggles 10 can communicate wirelessly with the Wi-Fi router 8. The VR goggles 10 and the VR control controller 11 can communicate wirelessly using Bluetooth or similar. The PC 17 and VR goggles 10 are also connected via Bluetooth or similar. A virtual meeting application is running on User D's PC 17, communicating with the virtual meeting server 5. Multiple users may participate in the meeting wearing VR goggles 10.

[0109] User E is at home (or a satellite office, etc.), and a Wi-Fi router 9 is installed at home. User E is using a PC 19, VR goggles 12, and a VR control controller 13. The Wi-Fi router 9 at User E's home can connect to communication network N. User E's PC 19 and VR goggles 12 can communicate wirelessly with the Wi-Fi router 9. The VR goggles 12 and the VR control controller 13 can communicate wirelessly using Bluetooth or similar. The PC 19 and VR goggles 12 are also connected via Bluetooth or similar. A virtual meeting application is running on User E's PC 19, communicating with the virtual meeting server 5. Multiple users may participate in the meeting wearing VR goggles 12.

[0110] Note that VR goggles may also be head-mounted displays such as AR (Augmented Reality) goggles or MR (Mixed Reality) goggles.

[0111] Content entered on the electronic whiteboard 2 at site 1 is sent to the virtual meeting server 5. The virtual meeting server 5 writes this content to the virtual whiteboard 18a and also sends the video of the virtual meeting space 18 to the VR goggles 10 and 12. This synchronizes (shares) the drawing data on the electronic whiteboard 2 between site 1 and sites 2 and 3. Similarly, drawing data handwritten by a user on the virtual whiteboard 18a in the virtual meeting space 18 at site 2 is sent to the virtual meeting server 5. The virtual meeting server 5 writes this drawing data to the virtual whiteboard 18a and also sends it to the electronic whiteboard 2 and VR goggles 12. This synchronizes the drawing data on the virtual whiteboard 18a between site 2 and sites 1 and 3. Drawing data at site 3 is also synchronized in the same way.

[0112] Furthermore, the screen data from PC 17 at site 2 is sent to the virtual meeting server 5 and written to the virtual meeting space 18 as virtual PC screen data 18b. When user D performs a predetermined operation, the virtual PC screen data 18b is copied to the virtual whiteboard 18a. The virtual meeting server 5 sends the content of the virtual whiteboard 18a to the electronic whiteboard 2. This allows sites 1 and 2 to share the screen data from PC 17 at site 2.

[0113] <Example Hardware Configuration> <<VRゴーグル> > Figure 22 is a hardware configuration diagram of VR goggles 10 and 12. VR goggles 10 and 12 have a CPU 80, main memory 81, ROM 82, display controller 86, wireless LAN controller 88, audio codec 90, and video codec 93, all connected via a bus 94.

[0114] The CPU 80 executes and processes the OS and control processing programs read from the ROM 82 into the main memory 81. The main memory 81 contains DRAM and is used as the CPU 80's work area, etc.

[0115] ROM82 is pre-programmed with the OS, the system program for when the power is turned on, and programs to control the VR goggles 10 and 12.

[0116] The CPU 80 is connected to the UART 83. The UART 83 is an interface for serial data exchange between the CPU 80 and the Bluetooth module 84, and consists of components such as a FIFO (First In, First Out) and a shift register.

[0117] The Bluetooth module 84 consists of an RF section and a baseband section, and is connected to an antenna 85, performing wireless communication in accordance with the Bluetooth standard.

[0118] The display controller 86 performs D / A conversion of character data, graphic data, image data, etc., and controls the display of this data on the LCD 87. The LCD 87 is a display unit that displays information.

[0119] The wireless LAN controller 88 executes a communication protocol compliant with the IEEE (Institute of Electrical and Electronics Engineers) 802.11ax standard and controls communication with other devices by transmitting and receiving radio waves via the antenna 89.

[0120] The audio signal input from microphone 91 is converted into audio data by an A / D (Analog / Digital) conversion circuit, and this audio data is encoded in the AAC (Advanced Audio Coding) format by audio codec 90. Furthermore, the AAC encoded data received from an external device is decoded by audio codec 90, and the audio, converted to analog by a D / A conversion circuit, is output from speaker 92. The video codec 93 decodes compressed video data received from an external device (for example, a format compliant with ITU-T Recommendation H.264). Data is exchanged between each of the above components via bus 94.

[0121] <<VR Operation Controller>> Figure 23 is a hardware configuration diagram of VR operation controllers 11 and 13. The VR operation controllers 11 and 13 have a UART 117, a main memory 111, a ROM 112, a six-axis acceleration / angular velocity sensor 113, a menu display button 114, a pointer display button 115, and an enter button 116, which are connected to a CPU 110.

[0122] The CPU 110 executes and processes a control processing program read from the ROM 112 into the main memory 111. The main memory 111 has a DRAM and is used as a work area of the CPU 110 and the like.

[0123] In the ROM 112, a system program at power-on, a program for transmitting the press information of the menu display button 114, the pointer display button 115, and the enter button 116 via Bluetooth is pre-written.

[0124] The six-axis acceleration / angular velocity sensor 113 outputs measurement data of acceleration and angular velocity. The UART 117 is an interface for exchanging serial data between the CPU 110 and a Bluetooth module 118, and is composed of a FIFO, a shift register, and the like. The Bluetooth module 118 is composed of an RF part and a baseband part, and an antenna 119 is connected thereto, and performs wireless communication compliant with the Bluetooth standard.

[0125] <<Virtual Conference Server>> Figure 24 shows the hardware configuration of an example of a virtual conference server 5. As shown in Figure 24, the virtual conference server 5 is built by a computer 500 and includes a CPU 501, ROM 502, RAM 503, HD (Hard Disk) 504, HDD (Hard Disk Drive) controller 505, display 506, external device connection I / F (Interface) 508, network I / F 509, bus line 510, keyboard 511, optical drive 514, and media I / F 516.

[0126] Of these, the CPU 501 controls the operation of the entire virtual conference server 5. The ROM 502 stores programs used to start the computer 500, such as the IPL. The RAM 503 is used as the work area for the CPU 501. The HD 504 stores various data such as programs. The HDD controller 505 controls the reading or writing of various data to the HD 504 according to the control of the CPU 501. The display 506 displays various information such as cursors, menus, windows, characters, or images. The external device connection I / F 508 is an interface for connecting various external devices. In this case, external devices include, for example, USB (Universal Serial Bus) memory and printers. The network I / F 509 is an interface for data communication using a network. The bus line 510 is an address bus, data bus, etc., for electrically connecting each component such as the CPU 501 shown in Figure 24.

[0127] The keyboard 511 is a type of input means equipped with multiple keys used for inputting characters, numbers, or various instructions. The optical drive 514 controls the reading or writing of various data to the optical storage medium 513, which is an example of a removable recording medium. The optical storage medium 513 may be a CD, DVD, Blu-ray (registered trademark), etc. The media interface 516 controls the reading or writing (storage) of data to the recording medium 515, such as flash memory.

[0128] <About the features> <<Virtual meeting server and interactive whiteboard>> Figure 25 is a functional block diagram that explains the functions of the virtual meeting server 5 and the electronic whiteboard 2 by dividing them into blocks.

[0129] <<Electronic Whiteboard>> The electronic whiteboard 2 includes a content receiving unit 21, a display control unit 22, a contact position detection unit 24, a data storage unit 25, an operation reception unit 26, a content transmission unit 27, a drawing data generation unit 28, a content management unit 29, a LAN communication control unit 41, a Bluetooth communication control unit 42, a device information transmission unit 43, a virtual conference connection unit 44, and a swipe information transmission unit 45. Each of these parts of the electronic whiteboard 2 is a function or means of function realized by any of the components shown in Figure 3 operating according to instructions from the CPU 401 following a program (virtual conference application) deployed from the SSD 404 to the RAM 403. The functions of the contact position detection unit 24, the drawing data generation unit 28, the data storage unit 25, the content transmission unit 27, and the content receiving unit 21 may be the same as in the first embodiment.

[0130] The display control unit 22 controls the display 480 to display content input to the electronic whiteboard 2 and content transmitted from the virtual meeting server 5.

[0131] The LAN communication control unit 41 is a means for sending and receiving data other than content with the virtual conference server 5. Specifically, the LAN communication control unit 41 is connected to a communication network N such as Ethernet (registered trademark) and performs data transmission and reception with other devices via the communication network N.

[0132] The Bluetooth communication control unit 42 performs communication in accordance with the Bluetooth standard. The virtual conference connection unit 44 uses the pre-configured URL and conference ID of the virtual conference service to control the connection to the virtual conference service running on the virtual conference server 5.

[0133] The device information transmission unit 43 transmits device information to the virtual conference server 5 and receives authentication from the virtual conference server 5 to participate in the conference.

[0134] When a user performs a swipe operation on the electronic whiteboard 2 to swipe the display area 251, the swipe information transmission unit 45 transmits the direction and size of the swipe to the virtual meeting server 5 via the LAN communication control unit 41.

[0135] <<Virtual meeting server>> The virtual meeting server 5 includes an object extraction unit 51, a content transmission unit 52, a content reception unit 53, a virtual meeting control unit 54, a whiteboard generation and management unit 55, a synchronization control unit 56, a communication control unit 57, a meeting data storage unit 58, a data storage unit 59, an aggregation processing unit 60, a type determination unit 61, a drawing data generation unit 62, a virtual display control unit 63, a pointer position calculation unit 64, a user authentication unit 65, a user information storage unit 66, and a content management unit 67. Each of these units of the virtual meeting server 5 is a function or means of functioning, realized by any of the components shown in Figure 24 operating according to instructions from the CPU 501 in accordance with a program expanded from the HD 504 to the RAM 503. Of these, the content transmission unit 52, content reception unit 53, data storage unit 59, aggregation processing unit 60, type determination unit 61, drawing data generation unit 62, and content management unit 67 may be the same as the functions described in the electronic whiteboard 2 in the first embodiment.

[0136] The object extraction unit 51 determines the boundaries of objects from the data added to the virtual whiteboard 18a and extracts objects. An object is a display that can be separated within any area, such as drawing data, icons, text data, or shapes. The object extraction unit 51 also groups objects if they are drawing data. Grouping can be done, for example, by grouping all drawing data within one screen into one group, or by grouping based on the distance between drawing data.

[0137] The content transmission unit 52 transmits the content sent from PCs 17 and 19 to the electronic whiteboard 2. The content transmission unit 52 then transmits the content received from the electronic whiteboard 2 to the VR goggles 10 and 12.

[0138] The content receiving unit 53 receives content transmitted from PCs 17 and 19. The content receiving unit 53 also receives content transmitted from the electronic whiteboard 2.

[0139] The virtual meeting control unit 54 stores the meeting participants and the devices connecting to the meeting (PCs 17, 19, VR goggles 10, 12) in the meeting data storage unit 58, associating them with the meeting identification information (meeting ID). The virtual meeting control unit 54 then conducts a remote meeting with each device for each meeting ID. The virtual meeting control unit 54 also stores virtual whiteboard screen data and audio data received from the devices connecting to the virtual meeting in the meeting data storage unit 58.

[0140] The whiteboard generation and management unit 55 generates and manages virtual whiteboards 18a within the virtual meeting space 18.

[0141] The synchronization control unit 56 is a means for synchronizing the content in the memory of the electronic whiteboard 2 and the content of the virtual whiteboard 18a so that they become the same. Specifically, the synchronization control unit 56 has the function of detecting updates to the content in the memory of the electronic whiteboard 2 and the content of the virtual whiteboard 18a. The synchronization control unit 56 has the function of reflecting the updated content in the memory of the electronic whiteboard 2 to the virtual whiteboard 18a. The synchronization control unit 56 has the function of reflecting the updated content of the virtual whiteboard 18a to the content of the electronic whiteboard 2.

[0142] The communication control unit 57 performs data transmission and reception between the electronic whiteboard 2, PCs 17 and 19, and VR goggles 10 and 12.

[0143] The conference data storage unit 58 stores virtual whiteboard data 18a, virtual PC screen data 18b, 18c, and audio data received from devices connected to the virtual conference.

[0144] The drawing data generation unit 62 generates menus displayed in the virtual meeting space 18 and pointers instructed to be displayed by the VR operation controller 11. The drawing data generation unit 62 also generates drawing data from the pointer trajectories (sequences of coordinate points) transmitted from the VR operation controllers 11 and 13, and generates shapes such as circles and rectangles.

[0145] The virtual display control unit 63 is a means for displaying screen data from PCs 17 and 19 as virtual PC screen data 18b and 18c in the virtual meeting space 18. The virtual display control unit 63 further displays menus, pointers, etc. in the virtual meeting space 18.

[0146] The pointer position calculation unit 64 calculates the position of the pointer (in the virtual meeting space 18) indicated by the orientation and movement of the VR operation controllers 11 and 13.

[0147] The user authentication unit 65 authenticates users participating in the virtual meeting by comparing the username and password received via the communication control unit 57 with the username and password stored in the user information storage unit 66.

[0148] The user information storage unit 66 stores the usernames and passwords of users participating in the virtual meeting.

[0149] Figure 26 is an example of a functional block diagram that explains the functions of the virtual meeting server 5, PCs 17 and 19, VR goggles 10 and 12, and VR operation controllers 11 and 13 by dividing them into blocks.

[0150] < <pc>> PCs 17 and 19 have a virtual conference connection unit 71, a remote desktop control unit 72, a wireless LAN communication control unit 73, a Bluetooth communication control unit 74, a display control unit 75, and a data storage unit 76. Each of these parts of PCs 17 and 19 is a function or means of functioning that is realized by any of the components shown in Figure 4 operating according to instructions from the CPU 521 in accordance with a program (virtual conference application) deployed from HD 524 to RAM 523.

[0151] The virtual meeting connection unit 71 connects to the virtual meeting service of the virtual meeting server 5 and transmits the username and password, which are authentication data. The virtual meeting connection unit 71 also manages the URL and meeting ID of the virtual meeting space 18 received from the virtual meeting service, and connects to the URL of the virtual meeting space 18 to perform the process of joining the virtual meeting. Furthermore, the virtual meeting connection unit 71 transmits the URL and meeting ID of the virtual meeting space 18 to the VR goggles 10 and 12 via the Bluetooth communication control unit 74.

[0152] The remote desktop control unit 72 sends the screen data from PCs 17 and 19 to the virtual meeting service in order to display it in the virtual meeting space 18.

[0153] The wireless LAN communication control unit 73 performs communication with the virtual conference server 5 via the Wi-Fi routers 8 and 9.

[0154] The Bluetooth communication control unit 74 performs communication with the VR goggles 10 and 12 in accordance with the Bluetooth standard. The display control unit 75 controls the display of screen data on the display (LCD).

[0155] The data storage unit 76 stores the address (URL) of the virtual meeting service provided by the virtual meeting server 5, the URL of the virtual meeting space 18 received from the virtual meeting service, the meeting ID, the usernames of the users using PCs 17 and 19, and passwords, etc.

[0156] <<VRゴーグル> > The VR goggles 10 and 12 include a display control unit 100, a wireless LAN communication control unit 101, a Bluetooth communication control unit 102, a virtual conference connection unit 103, a button information transfer unit 104, and a data storage unit 105. Each of these parts of the VR goggles 10 and 12 is a function or means of functioning, which is realized by any of the components shown in Figure 22 operating according to instructions from the CPU 80 in accordance with a program (virtual conference application) deployed from the ROM 82 to the main memory 81.

[0157] The display control unit 100 controls the display of the virtual whiteboard 18a and virtual PC screen data 18b and 18c transmitted from the virtual meeting service on the LCD 87. The wireless LAN communication control unit 101 performs communication with the virtual meeting server 5 via the Wi-Fi routers 8 and 9.

[0158] The Bluetooth communication control unit 102 performs communication with the VR operation controllers 11 and 13 in accordance with the Bluetooth standard. The virtual conference connection unit 103 uses the URL and conference ID of the virtual conference service transmitted from the PCs 17 and 19 to control the connection to the virtual conference service running on the virtual conference server 5.

[0159] The button information transfer unit 104 receives button press information from the VR operation controllers 11 and 13 and transmits information about the pressed button to the virtual meeting server 5 via the wireless LAN communication control unit 101.

[0160] The data storage unit 105 stores the address (URL) of the virtual meeting service provided by the virtual meeting server 5, the meeting ID, the user name of the user using the VR goggles 10, the password, and the like.

[0161] <<VR operation controller>> The VR operation controllers 11 and 13 have a Bluetooth communication control unit 120, a button information transmission unit 121, and an orientation / movement detection unit 122. These functions of the VR operation controllers 11 and 13 are functions or means realized by operating according to commands from the CPU 110 developed from the ROM 112 shown in FIG. 23 to the main memory 111.

[0162] The Bluetooth communication control unit 120 performs communication compliant with the Bluetooth standard with the VR goggles 10 and 12. The button information transmission unit 121 passes the press information of the menu display button 114, the pointer display button 115, and the confirmation button 116 to the Bluetooth communication control unit 120.

[0163] The orientation / movement detection unit 122 detects the orientation (posture in three-dimensional space) and movement of the VR operation controllers 11 and 13 from the acceleration and angular velocity data obtained from the six-axis acceleration / angular velocity sensor 113, and passes this information to the Bluetooth communication control unit 120.

[0164] <User's participation in the meeting> Next, the process when the user participates in the meeting will be described. Users A to C participate in the meeting from the meeting room (base 1). Users A to C participate in the meeting using the electronic blackboard 2. When users A to C in the meeting room (base 1) touch the "Meeting Participation" button displayed on the electronic blackboard 2, the LAN communication control unit 41 of the electronic blackboard 2 connects to the virtual meeting server 5, and the device information transmission unit 43 transmits the device information.

[0165] When the virtual meeting server 5 authenticates the electronic whiteboard 2 based on its device information, it determines whether there are any meetings registered that the electronic whiteboard 2 can join within a certain time frame from the present. If there are meetings that the electronic whiteboard 2 can join, the virtual meeting server 5 grants the electronic whiteboard 2 permission to join these meetings. Permission to join means that drawing data can be shared with other locations. Subsequently, the synchronization control unit 56 of the virtual meeting server 5 starts the synchronization process between the electronic whiteboard 2 and the virtual whiteboard 18a.

[0166] User D participates in the meeting from home (location 2). User D participates in the meeting wearing VR goggles 10 (in a visual state as if in the virtual meeting space). When User D launches the virtual meeting application on PC 17, the virtual meeting connection unit 44 connects to the virtual meeting service of the virtual meeting server 5 and displays a screen for entering the username and password. The URL of the virtual meeting service of the virtual meeting server 5 is pre-registered on PC 17 as setting data for the virtual meeting application. When User D enters the username and password, the virtual meeting connection unit 44 sends this data to the virtual meeting server 5. The virtual meeting server 5 authenticates this user and sends an authentication OK response to PC 17.

[0167] The virtual meeting application on PC17 displays a "Join Virtual Meeting" button. When User D selects the "Join" button, it displays a screen for entering the URL and meeting ID of the virtual meeting space 18. When User D enters the URL and meeting ID of the virtual meeting space 18 and presses the OK button, the virtual meeting connection unit 44 sends a connection command including this meeting ID to the URL of the virtual meeting space 18.

[0168] Virtual meeting server 5 determines if there are any meetings registered that user D can join within a certain time frame from now. If there are any meetings that user D can join, virtual meeting server 5 allows the connection between PC 17 and virtual meeting space 18.

[0169] Next, the virtual meeting application on PC17 sends the URL and meeting ID of the virtual meeting space 18 to user D's VR goggles 10 via Bluetooth. Upon receiving this data, the VR goggles 10 sends a connection command including this meeting ID to the URL of the virtual meeting space 18 via the virtual meeting connection unit 103. The virtual meeting service on the virtual meeting server 5 compares this meeting ID with the meeting ID of the meeting permitted on PC17, and upon confirmation of a match, authorizes the connection between the VR goggles 10 and the virtual meeting space 18.

[0170] When the virtual meeting application on PC17 connects to the virtual meeting space 18, it displays a remote desktop selection button. When user D selects this button, the remote desktop control unit 72 sends the screen data of PC17 to the virtual meeting server 5, which displays it in the virtual meeting space 18 as virtual PC screen data 18b. Subsequently, the virtual meeting service on the virtual meeting server 5 sends the updated display data of the virtual meeting space 18 to the VR goggles 10, which then displays this data on the LCD 87. The virtual display control unit 63 of the virtual meeting server 5 controls the virtual display data 18b of PC17 displayed in the virtual meeting space 18 so that it is only displayed on user D's VR goggles 10.

[0171] The process for user E, who is at home (location 3), to join the virtual meeting is the same as for user D.

[0172] Figure 27 shows an example of the virtual meeting space 18 displayed on user D's VR goggles 10. As shown in Figure 27, the virtual meeting space 18 displays a virtual whiteboard 18a and virtual PC screen data 18b. Similarly, user E's VR goggles 12 display a virtual whiteboard 18a and virtual PC screen data 18c. The virtual whiteboard 18a does not have a menu bar 254, but it may have one.

[0173] User D gripped the VR control controller 11 with the grip portion (the part held by the hand) facing upwards. The VR control controller 11 transmits output data (information on orientation and movement) from the 6-axis accelerometer / angular velocity sensor 113 to the virtual meeting server 5 via the VR goggles 10. The virtual meeting server 5 determines from this output data that the VR control controller 11 is upside down. The whiteboard generation and management unit 55 of the virtual meeting server 5 then sets the virtual whiteboard 18a to handwriting mode and displays the position of the VR control controller 11 at that moment as a pen mark in the upper left corner of the virtual whiteboard 18a. In this state, when User D moves the VR control controller 11 in a plane perpendicular to the ground, the pen mark also moves in that direction. Then, when User D moves the VR control controller 11 a predetermined distance in the depth direction (away from User D's body: in the horizontal plane relative to the ground), the drawing data generation unit 62 determines that the pen is down. Subsequently, when user D moves the VR control controller 11 in a plane perpendicular to the ground and performs a handwriting motion, the drawing data generation unit 62 generates drawing data from the trajectory of that movement. The drawing data generation unit 62 writes the drawing data to the virtual whiteboard 18a. When user D moves the VR control controller 11 a predetermined distance toward the user (towards user D's body: in a horizontal plane relative to the ground), the drawing data generation unit 62 puts the pen up.

[0174] In this way, user D can handwrite characters on the virtual whiteboard 18a using the VR operation controller 11. The synchronization control unit 56 of the virtual meeting server 5 transmits this drawing data to the electronic whiteboard 2, and the electronic whiteboard 2 displays this drawing data on the display 480.

[0175] <Action or process> Figures 28 and 29 are sequence diagrams illustrating the process of drawing data using the VR operation controller 11 and drawing data on the electronic whiteboard 2. Note that in the explanation of Figure 28, it is assumed that the electronic whiteboard 2 and user D are already participating in the meeting.

[0176] S101: User D holds the VR control controller 11 with the grip portion (the part held in the hand) facing upwards, and moves the VR control controller 11 a predetermined distance in the depth direction (away from User D's body: in the horizontal plane relative to the ground) (pen down). User D moves the VR control controller 11 in the vertical plane relative to the ground and performs a handwriting action.

[0177] S102: The orientation and motion detection unit 122 of the VR operation controller 11 detects the orientation (posture in 3D space) and motion of the VR operation controller 11 from the acceleration and angular velocity data obtained from the 6-axis acceleration and angular velocity sensor 113, and the Bluetooth communication control unit 120 transmits this information to the VR goggles 10.

[0178] S103: The Bluetooth communication control unit 102 of the VR goggles 10 receives orientation and movement information from the VR operation controller 11, and the wireless LAN communication control unit 101 transmits this information to the virtual meeting server 5.

[0179] S104: The communication control unit 57 of the virtual meeting server 5 receives this information, and the pointer position calculation unit 64 calculates the pointer's position. When the drawing data generation unit 62 determines that the orientation (posture in 3D space) of the VR operation controller 11 is such that the grip portion (the part held by the hand) is facing upwards, it generates drawing data based on the trajectory of the pointer's position. The drawing data generation unit 62 writes the drawing data to the virtual whiteboard 18a.

[0180] S105: The virtual display control unit 63 of the virtual meeting server 5 updates the display on the virtual whiteboard 18a, and the communication control unit 57 transmits this display data to the VR goggles 10.

[0181] S106: The content transmission unit 52 of the virtual conference server 5 transmits this drawing data to the electronic whiteboard 2 via the communication control unit 57.

[0182] S107: The wireless LAN communication control unit 101 of the VR goggles 10 receives display data, and the display control unit 100 displays it.

[0183] S108: The content receiving unit 21 of the electronic whiteboard 2 receives drawing data, and the display control unit 22 adds the drawing data to the current screen data to display it on the display 480.

[0184] S109~S116: While user D is writing by hand on the VR control controller 11, the same process is repeatedly executed.

[0185] S117: When user D selects the remote desktop selection button displayed by the virtual meeting application on PC17, the remote desktop control unit 72 sends the screen data from PC17 to the virtual meeting server 5.

[0186] S118: The communication control unit 57 of the virtual conference server 5 receives this, and the virtual display control unit 63 places the virtual PC screen data 18b in the virtual conference space 18.

[0187] S119: The virtual conference server 5 transmits virtual PC screen data 18b to the VR goggles 10 via the communication control unit 57.

[0188] Refer to Figure 29 to explain the synchronization when handwritten on the electronic whiteboard 2. S121: User B (or A or C) in the conference room uses the electronic pen 490 to write or display shapes in the display area 251.

[0189] S122: The contact position detection unit 24 detects the contact position of the electronic pen 490, and the drawing data generation unit 28 generates and displays drawing data and figures.

[0190] S123: The content transmission unit 27 of the electronic whiteboard 2 sequentially transmits this drawing data to the virtual conference server 5.

[0191] S124: The content receiving unit 53 of the virtual conference server 5 receives drawing data, and the synchronization control unit 56 writes the drawing data to the virtual whiteboard 18a.

[0192] S125: The virtual meeting server 5 transmits the display data of the virtual whiteboard 18a to the home VR goggles 10 and 12 via the communication control unit 57.

[0193] S126: The wireless LAN communication control unit 101 of the VR goggles 10 receives display data from the virtual whiteboard 18a, and the display control unit 100 displays it.

[0194] S127~S132: While User B writes on the electronic whiteboard 2, the same process is repeatedly executed.

[0195] <Aggregated display of virtual whiteboard and virtual PC screen data> As a result of the above processing, the virtual whiteboard 18a, virtual PC screen data 18b, and 18c are displayed in the virtual meeting space 18.

[0196] Figure 30 shows an example of the display on the electronic whiteboard 2 (Figure 30(a)) and an example of the display of the virtual meeting space 18 as viewed by user D (Figure 30(b)).

[0197] When content is added to the virtual whiteboard 18a, the object extraction unit 51 analyzes the shape of the boundary (edge) between displayed pixels (pixel data values ​​other than zero) and non-displayed pixels (pixel data values ​​of zero) for that content and determines that only characters (drawing data) are displayed on the virtual whiteboard 18a. The object extraction unit 51 then groups this character data (drawing data).

[0198] Next, the type determination unit 61 determines that the grouped character data is "drawn (handwritten)". The determination method may be the same as in the first embodiment.

[0199] Next, the content management unit 67 stores the coordinates of the top left corner of the rectangular area circumscribing this group (the coordinates of the virtual meeting space) and the size of the rectangular area (the number of display pixels in the horizontal and vertical directions) along with the type of content (hand-drawn data) in the data storage unit 59.

[0200] Here, the coordinates in the virtual meeting space 18 are expressed using a display coordinate system where the pixel position at the top-left corner of the virtual whiteboard 18a is the origin, with the right direction being the positive X-axis and the downward direction being the positive Y-axis. That is, Display coordinates = (number of pixels in the X-axis direction from the pixel at the origin, number of pixels in the Y-axis direction from the pixel at the origin) That is the case.

[0201] Next, the screen data from user D's PC 17 was to be displayed on the virtual whiteboard 18a so that other users could also see it. To do this, user A, who is in the conference room, moves the content 211 displayed on the electronic whiteboard 2 outside the display area 251 of the electronic whiteboard 2. When user A performs a leftward swipe operation on the display of the electronic whiteboard 2, the electronic whiteboard 2 determines that a swipe operation has occurred because the distance between the two coordinates detected by the contact position detection unit 24 during the time interval in which coordinates are detected exceeds a predetermined value. The display control unit 22 of the electronic whiteboard 2 then erases the content 211 (handwritten drawing data in Figure 30) in the memory for display, and the display 480 of the electronic whiteboard 2 returns to a blank state. The swipe information transmission unit 45 of the electronic whiteboard 2 transmits the swipe operation information (direction and magnitude (vector information)) to the virtual conference server 5.

[0202] The virtual display control unit 63 of the virtual meeting server 5 moves the content 211 to the left within the virtual meeting space 18 according to the direction and size of the swipe operation.

[0203] Figure 31 shows the virtual meeting space 18 with content 211 moved to the left. As shown in Figure 31, the virtual meeting space 18 displays content 211 (grouped drawing data), a virtual whiteboard 18a, and virtual PC screen data 18b separately.

[0204] Figure 32 shows the content management table managed by the content management unit 67. This content management table contains information on the grouped content 211. The information managed can be the same as in Figure 10.

[0205] Figure 33 shows the virtual PC screen data 18b of the virtual meeting space 18. Content 212 is displayed on the virtual PC screen data 18b. A movement bar 221 is attached to the top edge of the virtual PC screen data 18b. When user D presses the pointer display button 115 on the VR operation controller 11 and places the pointer over this movement bar 221 and holds it down, the virtual PC screen data 18b becomes movable.

[0206] When user D moves the pointer while pressing the pointer display button 115, the virtual PC screen data 18b also moves accordingly. When user D overlays the virtual PC screen data 18b onto the virtual whiteboard 18a with this operation, the virtual display control unit 63 copies the virtual PC screen data 18b to the virtual whiteboard 18a. Subsequently, user D moves the pointer, which is superimposed on the movement bar, while pressing the pointer display button 115 on the VR operation controller 11, in the same manner as above, to move the virtual PC screen data 18b away from the virtual whiteboard 18a. This operation is performed to display the virtual PC screen data 18b again in the virtual meeting space 18, but it is unnecessary if the setting is such that the original virtual PC screen data 18b remains even after it is copied to the virtual whiteboard 18a.

[0207] The synchronization control unit 56 of the virtual meeting server 5 detects an update to the virtual whiteboard 18a. The synchronization control unit 56 transmits the content of the virtual whiteboard 18a to the electronic whiteboard 2 via the communication control unit 57. The electronic whiteboard 2 displays the received content of the virtual whiteboard 18a.

[0208] Figure 34 shows the content 212 of the virtual whiteboard 18a displayed by the electronic whiteboard 2. Figure 35 shows the content 211, 212, and virtual PC screen data 18b present in the virtual meeting space 18. As shown in Figure 34, the electronic whiteboard 2 was able to display the same virtual PC screen data 18b as the screen data of PC 17. Therefore, users A-C and user D in the meeting room were able to share the content 212 of the virtual PC screen data 18b. Also, as shown in Figure 35, the content 212 of the virtual PC screen data 18b was copied to the virtual whiteboard 18a.

[0209] Since content 212 has been added to the virtual whiteboard 18a, the object extraction unit 51 analyzes the shape of the boundary (edge) between displayed pixels (pixel data values ​​other than zero) and non-displayed pixels (pixel data values ​​of zero) for this content 212 and determines that text and images are displayed on the virtual whiteboard 18a. Then, the object extraction unit 51 groups the text data 231.

[0210] Next, the virtual meeting service passes this grouped character data 231 to the type determination unit 61. The type determination unit 61 uses a type determination model to determine that this character data is "text".

[0211] Next, the content management unit 67 stores the coordinates of the top left corner of the rectangular area of ​​this group of text data (coordinates of the virtual meeting space) and the size of the group's rectangular area (number of display pixels in the horizontal and vertical directions) along with the type of content (text data) in the data storage unit 59 (content management table).

[0212] Next, the object extraction unit 51 determines that there are a circular object 235, three objects (trash can 232, folder 233, document 234), and a rectangular object 236 in the area other than the text. The virtual meeting service then passes the circular object 235 to the type determination unit 61. The type determination unit 61 uses a type determination model to determine that this object 235 is a "graph (image)". The content management unit 67 then stores the coordinates of the top left corner of the rectangular area circumscribing the circular object 235 and the size of this rectangular area (number of display pixels in the horizontal and vertical directions) along with the content type (graph (image)) in the data storage unit 59 (content management table).

[0213] Next, the virtual meeting service passes the following object data (trash can 232) to the type determination unit 61. The type determination unit 61 attempts to determine the type using the type determination model, but since the trash can 232 has not been trained, it determines that this object is "other". The content management unit 67 stores the coordinate and size information of objects classified as "drawing (handwritten)", "shape", "text", "photo (image data)", and "graph (image data)" in the data storage unit 59 (content management table), but does not store the data of other objects in the data storage unit 59 (content management table).

[0214] The virtual meeting service then passes the next object data (folder 233) to the type determination unit 61, but the type determination unit 61 determines it to be "other," so the content management unit 67 does not store this object in the data storage unit 59 (content management table).

[0215] The virtual meeting service then passes the next object data (document 234) to the type determination unit 61, but the type determination unit 61 determines it to be "other," so the content management unit 67 does not store this object in the data storage unit 59.

[0216] The virtual meeting service then passes the rectangular object 236 to the type determination unit 61. The type determination unit 61 determines that this object is a "graphic". The content management unit 67 then stores the coordinates of the top left corner of the rectangular area circumscribing this rectangular object, and the size of this rectangular area (number of display pixels in the horizontal and vertical directions), along with the content type (graphic), in the data storage unit 59 (content management table).

[0217] As described above, the objects identified from content 212 are each registered as content in the content management table (see Figure 37).

[0218] Next, the user A hand-wrote drawing data on the electronic whiteboard 2 in the conference room and shared it with users at other locations. To return the display area 251 to a blank state, user A performed a swipe operation on the electronic whiteboard 2. User A moved the content 212 of the virtual whiteboard 18a displayed on the electronic whiteboard 2 outside the display area 251 of the electronic whiteboard 2. When user A performed an upward swipe operation on the display of the electronic whiteboard 2, the distance between the two coordinates detected by the contact position detection unit 24 of the electronic whiteboard 2 became greater than a predetermined value during the time interval in which the coordinates were detected, and the unit determined that a swipe operation had occurred. The display control unit 22 of the electronic whiteboard 2 erased the content 212 in the memory for display, and the display 480 of the electronic whiteboard 2 returned to a blank state.

[0219] The swipe information transmission unit 45 of the electronic whiteboard 2 transmits the information of the swipe operation (direction and magnitude (vector information)) to the virtual meeting server 5. Upon receiving this information, the virtual display control unit 63 moves the content 212 of the virtual whiteboard 18a upwards within the virtual meeting space according to the direction and magnitude of the movement.

[0220] Figure 36 shows the virtual conference space 18 where the content 212 on the virtual whiteboard 18a has been moved upward. As shown in Figure 36, in the virtual conference space 18, the content 211 (grouped drawing data), the content 212 (content of the virtual whiteboard after swiping), the virtual whiteboard 18a, and the virtual PC screen data 18b are displayed separately.

[0221] Figure 37 shows the content management table managed by the content management unit 67 in the virtual conference space 18 of Figure 36. The number of objects whose types are determined in the content 212 is three (text data, graph, figure). Therefore, in addition to the content 211, a total of four contents are registered in the content management table. The content with the content ID of "1" corresponds to the content 211, the content with the content ID of "2" corresponds to the character data 231, the content with the content ID of "3" corresponds to the circular object 235, and the content with the content ID of "4" corresponds to the rectangular object 236.

[0222] In addition, in Figure 37, one screen data is decomposed into multiple contents, but the content management unit 67 may register one screen data as it is in the content management table. In this case, the type is the screen data or the like. The aggregation processing unit 60 may aggregate the virtual PC screen data 18b and the content input to the electronic blackboard 2.

[0223] Subsequently, user A in the conference room writes by hand on the electronic blackboard 2. The content transmission unit 27 of the electronic blackboard 2 transmits the drawing data to the virtual conference server 5. When the virtual conference server 5 receives the drawing data, the synchronization control unit 56 displays the drawing data on the virtual whiteboard 18a.

[0224] FIG. 38 shows the content 215 handwritten on the electronic blackboard 2. This content 215 is drawing data. FIG. 39 shows the virtual whiteboard 18a in the virtual conference space 18 synchronized with the drawing data handwritten on the electronic blackboard 2. Compared with FIG. 36, in FIG. 39, the content 215 is newly displayed on the virtual whiteboard 18a.

[0225] The virtual conference service passes the content 215 on the virtual whiteboard 18a to the object extraction unit 51. The object extraction unit 51 analyzes the shape of the boundary (edge) between the display pixels (pixel data values are non-zero) and the non-display pixels (pixel data values are zero) for this content, and determines that only characters are displayed on the virtual whiteboard 18a. Then, the object extraction unit 51 groups this character data (drawing data).

[0226] Subsequently, the virtual conference service passes this grouped character data to the type determination unit 61. The type determination unit 61 uses the type determination model to determine that this character data is "drawing (handwritten)". Subsequently, the content management unit 67 stores the coordinates of the upper left corner of the rectangular area circumscribing this group of character data (coordinates in the virtual conference space) and the size of the rectangular area (number of display pixels in the horizontal and vertical directions) together with the type of the content (drawing data) in the content management table.

[0227] FIG. 40 shows the content management table managed by the content management unit 67 in the virtual conference space 18 of FIG. 39. Information on the content 215 added to the virtual whiteboard 18a in FIG. 39 is newly registered in this content management table. The content with the content ID of "5" corresponds to the content 215.

[0228] Here, User A, who is in the conference room, presses the aggregate display menu button 253 shown in Figure 6 to display the handwritten drawing data (content 211) previously created on the electronic whiteboard 2 together with the currently displayed drawing data (content 215). The electronic whiteboard 2 displays the condition setting screen 260, so User A selects "drawing (handwritten)" as the type and presses the aggregate button 263.

[0229] The content management unit 67 refers to the content management table and confirms that the content of type "drawing (handwritten)" has content IDs "1" and "5". Then, the aggregation processing unit 60 determines the placement of the content with content IDs "1" and "5" according to the content placement rules described in the first embodiment.

[0230] In this embodiment, a virtual meeting space 18 is used instead of the display extension area 252 shown in the first embodiment. As shown in Figure 39, a vertical perpendicular line 256 passing through the center point of the virtual meeting space 18 divides the virtual meeting space 18 into a left area and a right area, and determines whether the content exists only in the left area or only in the right area. Since both contents 211 and 215 with content IDs "1" and "5" exist in the left area, the aggregation processing unit 60 compares the Y coordinates of the starting coordinates (top-left coordinates) of both contents 211 and 215. Since content 215 with content ID "5" has a smaller value (is higher), the aggregation processing unit 60 places content 215 with content ID "5" on the upper side and content 211 with content ID "1" on the lower side.

[0231] Next, according to the content management table in Figure 40, the coordinates of content 215 with content ID "5" are (h5, v5). This indicates that there is empty space of h5 and v5 on the top and left sides of content 215, respectively. The aggregation processing unit 60 moves content 215 with content ID "5" to the left by h5 pixels and upward by v5 pixels. In other words, the aggregation processing unit 60 aligns the upper left corner of content 215 with the origin and fills in the empty space.

[0232] Next, the aggregation processing unit 60 moves content 211, whose content ID is "1", so that the coordinates of its upper left corner are (0, 920+a). a is the gap between content 211 and 215, for example, 20.

[0233] Figure 41 shows a virtual whiteboard 18a with content 215 at the top and content 211 at the bottom. The top left corner of content 215, which has content ID "5", coincides with the top left corner of the virtual whiteboard 18a. There is also a gap of 'a' between content 215, which has content ID "5", and content 211, which has content ID "1". On the other hand, the bottom of content 211, which has content ID "1", extends beyond the virtual whiteboard 18a.

[0234] In the state shown in Figure 41, content 211 with content ID "1" extends beyond the Y-axis direction (positive direction) of the virtual whiteboard 18a, and the electronic whiteboard 2 cannot display this extended portion. Therefore, the aggregation processing unit 60 shrinks content 215 with content ID "5" and content 211 with content ID "1" so that content 211 with content ID "1" also fits within the virtual whiteboard 18a.

[0235] Assuming that the vertical pixel count of virtual whiteboard 18a (excluding the menu bar and the area below it) is 1200, the vertical pixel count of content 215 and 211 plus the gap between them is 510 + 920 + 20, so the reduction ratio is as follows. Reduction rate=1200 / (510+920+20) The aggregation processing unit 60 reduces content 215 with content ID "5" and content 211 with content ID "1" by this reduction ratio.

[0236] The aggregation processing unit 60 may change the scaling factor according to the character size, even if the content of the drawing data is the same. The aggregation processing unit 60 calculates the average size of one character using an existing method and changes the scaling factor of content 211 and 215. For example, let A be the height of content 211, L1 be the character size, and r1 be the scaling factor; B be the height of content 215, L2 (>L1), r2 be the scaling factor; and T be the height of the display area. In this case, the scaling factors r1 and r2 can be determined from the following relationship. (L2 / L1)·r2·A+r2·B=T r1 = (L2 / L1)·r2 The aggregation processing unit 60 reduces content 211 by a reduction ratio r1 and content 215 by a reduction ratio r2, so that the font sizes become roughly the same and the two contents 211 and 215 can be scaled to fit within the display area. Note that the reduction ratio may become greater than 1 depending on the font size, so the aggregation processing unit 60 may also enlarge the content.

[0237] Figure 42(a) shows the reduced virtual whiteboard 18a. Content 215 with content ID "5" and content 211 with content ID "1" are contained within the virtual whiteboard 18a. As shown in Figure 42(b), the electronic whiteboard 2 receives the contents 211 and 215 from this virtual whiteboard 18a and can display content 215 with content ID "5" and content 211 with content ID "1" on the display 480.

[0238] <Action or process> Figure 43 is a sequence diagram illustrating the processing or operation performed by the virtual meeting system 700. Assume that at the start of Figure 43, the virtual meeting space 18 is as shown in Figure 35.

[0239] S133: User A performs a swipe operation on the electronic whiteboard 2. User A swipes the content displayed on the electronic whiteboard 2 upwards. The display control unit 22 of the electronic whiteboard 2 erases the handwritten drawing data stored in the memory for display, and the display 480 of the electronic whiteboard 2 returns to a blank state.

[0240] S134: The swipe information transmission unit 45 of the electronic blackboard 2 transmits the information on the swiped operation (direction and magnitude (vector information)) to the virtual conference server 5.

[0241] S135: When the virtual conference server 5 receives this information, the virtual display control unit 63 moves the content of the virtual whiteboard 18a upward within the virtual conference space 18 according to the direction and magnitude of the movement.

[0242] S136: The virtual conference server 5 transmits the updated display data within the virtual conference space 18 to the VR goggles 10 and the VR goggles 12.

[0243] S137: The wireless LAN communication control unit 101 of the VR goggles 10 receives this display data, and the display control unit 100 displays it.

[0244] S138: Subsequently, user A in the conference room writes by hand on the electronic blackboard 2.

[0245] S139: The content transmission unit 27 of the electronic blackboard 2 transmits the drawing data to the virtual conference server 5.

[0246] S140: When the content receiving unit 53 of the virtual conference server 5 receives drawing data, the synchronization control unit 56 adds the drawing data to the virtual whiteboard 18a. At this time, the object extraction unit 51 groups the drawing data, and the type determination unit 61 determines that the content type of this drawing data is drawing (handwritten). The content management unit 67 then stores the coordinates of the top left corner (coordinates of the virtual conference space) and the size of the rectangular area (number of display pixels in the horizontal and vertical directions) that circumsects this group of drawing data in the content management table, along with the content type (drawing data). When new drawing data is added near existing drawing data on the virtual whiteboard 18a, this drawing data is included in the same group as the existing drawing data. That is, while adding drawing data included in the same group to the virtual whiteboard 18a, only the size of the rectangular area circumsects this group (number of display pixels in the horizontal and vertical directions) is updated in the content management table.

[0247] S141: The virtual meeting server 5 sends the updated display data of the virtual whiteboard 18a to the VR goggles 10 and VR goggles 12.

[0248] S142: The wireless LAN communication control unit 101 of the VR goggles 10 receives this display data, and the display control unit 100 displays it.

[0249] S143~S147: While User A writes on electronic whiteboard 2, the same process is repeatedly executed.

[0250] S148: User A selects the content type and presses the aggregate button 263. S149: The LAN communication control unit 41 of the electronic whiteboard 2 sends an aggregation request of a specified type to the virtual conference server 5.

[0251] S150: When the communication control unit 57 of the virtual conference server 5 receives an aggregation request, the aggregation processing unit 60 extracts the content specified by type from the virtual conference space 18 (content management table), aggregates it, and places it on the virtual whiteboard 18a.

[0252] S151: The communication control unit 57 of the virtual meeting server 5 transmits one or more content items aggregated on the virtual whiteboard 18a to the electronic blackboard 2.

[0253] S152: The LAN communication control unit 41 of the electronic whiteboard 2 receives one or more content items, and the display control unit 22 causes the display 480 to display one or more content items.

[0254] S153: The virtual meeting server 5 sends the updated display data of the virtual whiteboard 18a to the VR goggles 10 and VR goggles 12.

[0255] S154: The wireless LAN communication control unit 101 of the VR goggles 10 receives this display data, and the display control unit 100 displays it.

[0256] <Main effects> According to this embodiment, content displayed by multiple devices in the virtual meeting space 18 can be consolidated and arranged in the display area 251 of the electronic whiteboard 2. Not only the content input to the electronic whiteboard 2, but also the content displayed by the electronic whiteboard 2 and the content displayed by PCs 17 and 19 can be consolidated and arranged in the display area 251, thus improving the ability to view the content at a glance.

[0257] [Third Embodiment] This embodiment describes an information provision method that displays the content centrally arranged in the first embodiment to multiple devices conducting remote meetings at different locations.

[0258] <Example System Configuration> Figure 44 shows an example of the system configuration of the display system 900 in this embodiment. In the display system 900 of Figure 44, the electronic whiteboard 801 and two tablet PCs 802 and 803 can communicate with the conference server 800 via a communication network. The electronic whiteboard 801 is located at location A, tablet PC 802 at location B, and tablet PC 803 at location C. The electronic whiteboard 801 and tablet PCs 802 and 803 may be the same as in the first embodiment. When the electronic whiteboard 801 and tablet PCs 802 and 803 are not distinguished, they will be referred to as "each device".

[0259] The conference server 800 is a server that conducts remote conferences. The conference server 800 receives image and audio data from locations A to C and transmits it to locations A to C, excluding the source location. The electronic whiteboard 801 and the two tablet PCs 802 and 803 display the received image data and output the audio data, respectively, enabling remote conferences between users in different locations.

[0260] In the configuration shown in Figure 44, the electronic whiteboard 801 and tablet PCs 802 and 803 are each displaying an arbitrary location within the display extension area 252. The top-left coordinates and area size of the display extension area 252 displayed by the electronic whiteboard 801 and tablet PCs 802 and 803 are always managed by the conference server 800. When a user of their own device or another device performs a predetermined operation, the content data of the display area 251 is displayed in the whiteboard area of ​​each device.

[0261] The hardware configuration of the electronic whiteboard 801 may be the same as in Figure 3. The hardware configuration of the tablet PCs 802 and 803 may be the same as in Figure 4. The hardware configuration of the conference server 800 may be the same as in Figure 24.

[0262] <About the features> <<Conference Server>> Figure 45 is a functional block diagram illustrating the functions of the conference server 800 in blocks. The conference server 800 includes a conference control unit 810, a conference data storage unit 811, a whiteboard control unit 812, a type determination unit 813 (type determination model), a content management unit 814, an aggregation processing unit 815, a display data generation unit 816, a communication control unit 817, a user authentication unit 818, a user information storage unit 819, a device authentication unit 820, and a device information storage unit 821. Each of these parts of the conference server 800 is a function or means of functioning, realized by any of the components shown in Figure 24 operating according to instructions from the CPU 501 in accordance with a program deployed from the HD 504 to the RAM 503. Of these, the aggregation processing unit 815, the type determination unit 813, and the content management unit 814 may be the same as the functions described in the electronic whiteboard 2 in the first embodiment. The communication control unit 817, conference data storage unit 811, user authentication unit 818, and user information storage unit 66 may have the same functions as described in the virtual conference server 5 in the second embodiment.

[0263] The conference control unit 810 stores information about conference participants in the conference data storage unit 811, associating it with conference identification information (conference ID), and then executes the conference application. For each conference ID, the conference application executes functions such as distributing audio data, image data (camera input video), conference materials, screen data, etc., received from conference participants' terminals (electronic whiteboard 801, tablet PC 802, tablet PC 803) to other terminals. The conference ID is information that identifies a conference and is generated when a conference reservation is made, or, if no conference reservation is made, when one of the tablet PCs 802, 803, or electronic whiteboard 801 connects to the conference server 800.

[0264] The meeting data storage unit 811 stores information about meeting participants, associating it with the meeting ID. The meeting data storage unit 811 also stores audio data, image data (camera input video), meeting materials data, etc., received from the tablet PC 802, tablet PC 803, and electronic whiteboard 801.

[0265] The whiteboard control unit 812 manages the positions of the display area displayed on the display 480 by the electronic whiteboard 801, and the display areas displayed on the display 526 by the tablet PCs 802 and 803 in the display extension area 252. The whiteboard control unit 812 also transmits data to allow any of the electronic whiteboard 801, tablet PC 802, or tablet PC 803 to display the data of the whiteboard area displayed on their displays on the displays of other devices.

[0266] The content type determination unit 813 determines the type of content. The content type determination unit 813 has a content type determination model that has been machine-trained using training data for determining the type of content. The content type determination model extracts features of the content using a feature extraction model suitable for images, such as a CNN, and determines the type of content.

[0267] The content management unit 814 registers the content and information received from the tablet PC 802, tablet PC 803, and electronic whiteboard 801 in the content management table (see Figure 10).

[0268] The aggregation processing unit 815 places the user-specified type of content within the display area of ​​the tablet PC 802, tablet PC 803, and electronic whiteboard 801, and also scales it down to fit within the display area. The aggregation processing unit 815 can also search for content that matches keywords.

[0269] The display data generation unit 816 generates display data that the meeting application will display on the tablet PC 802, tablet PC 803, and electronic whiteboard 801.

[0270] The communication control unit 817 executes the communication protocol of the conference application and also performs communication control as a web server.

[0271] The user authentication unit 818 authenticates users who will use the conference application by comparing the user ID and password received via the communication control unit 817 with the user ID and password stored in the user information storage unit 819.

[0272] The user information storage unit 819 stores the user ID and password of users who use the conference application.

[0273] The device authentication unit 820 authenticates the device using the conference application by comparing the device ID of the device (electronic whiteboard 801) received via the communication control unit 817 with the device ID stored in the device information storage unit 821.

[0274] <<Electronic Whiteboard 801>> Figure 46 is a functional block diagram illustrating the functions of the electronic whiteboard 801 divided into blocks. The electronic whiteboard 801 includes a conference control unit 830, an audio transmission / reception unit 831, a video transmission / reception unit 832, a whiteboard control unit 833, a contact position detection unit 834, a drawing data generation unit 835, an operation reception unit 836, a display control unit 837, a communication control unit 838, and a data storage unit 839. Each of these parts of the electronic whiteboard 801 is a function or means of functioning, realized by any of the components shown in Figure 3 operating according to instructions from the CPU 401 following a program (virtual conference application) deployed from the SSD 404 to the RAM 403. Of these, the contact position detection unit 834, the drawing data generation unit 835, the operation reception unit 836, the display control unit 837, and the data storage unit 839 may be the same as in the first embodiment.

[0275] The conference control unit 830 executes a conference application to conduct remote conferences with terminals (tablet PC 802, tablet PC 803) located at the site via the conference server 800.

[0276] The audio transmission / reception unit 831 encodes the audio data, which is A / D converted from the audio signal input from the microphone 440, using the AAC method and transmits it, or decodes the received encoded audio data and outputs the D / A converted audio signal to the speaker 450.

[0277] The video transmission / reception unit 832 encodes the video data, which is A / D converted from the video signal input from the camera 460, according to ITU-T Recommendation H.265 and transmits it, or decodes the received encoded video data and outputs it to the display control unit 837.

[0278] The whiteboard control unit 833 transmits handwritten drawing data and content such as diagrams and photographs added to the display area 251 to the display extension area 252 of the conference server 800. The whiteboard control unit 833 also controls the movement of the area to be displayed on the display 480 of the electronic whiteboard 801 within the display extension area 252 of the conference server 800.

[0279] The contact position detection unit 834 detects the contact position of the user's hand H or the electronic pen 490.

[0280] The drawing data generation unit 835 generates drawing data from a sequence of coordinate points input by the contact position detection unit 834 when the user writes by hand, and also generates shapes such as circles and rectangles.

[0281] The operation reception unit 836 receives operations on the electronic whiteboard 801. For example, it determines the location of a button or the like at the coordinates where the electronic pen 490 or hand H has made contact, and accepts the button press.

[0282] The display control unit 837 displays content such as drawing data and graphics input to the machine, displays meeting information, and displays screen data received from the meeting server 800.

[0283] The communication control unit 838 executes communication protocols such as TCP / IP, RTCP (RTP Control Protocol), and HTTP, and in wireless communication, it executes communication protocols compliant with the IEEE 802.11ax standard.

[0284] The data storage unit 839 stores handwritten drawing data, display screen data, the device ID of the electronic whiteboard 801, and the like.

[0285] <<Tablet PCs 802, 803>> Figure 47 is a functional block diagram that explains the functions of the tablet PCs 802 and 803 by dividing them into blocks. The tablet PCs 802 and 803 have a conference control unit 840, an audio transmission / reception unit 841, a video transmission / reception unit 842, a whiteboard control unit 843, a contact position detection unit 844, a drawing data generation unit 845, an operation reception unit 846, a display control unit 847, a communication control unit 848, and a data storage unit 849. Each of these parts of the tablet PCs 802 and 803 is a function or means of functioning that is realized by any of the components shown in Figure 4 operating according to instructions from the CPU 521 in accordance with a program deployed from the HD 524 to the RAM 523.

[0286] The functional units other than the data storage unit 849 are the same as those of the electronic whiteboard 801. The data storage unit 849 stores handwritten drawing data, display screen data, user IDs and passwords of each tablet PC user, etc.

[0287] <Flow of events leading up to the start of the meeting> Figure 48 is a sequence diagram illustrating the process by which an electronic whiteboard 801 and two tablet PCs 802 and 803 initiate a remote conference via a conference server 800.

[0288] S201: Users of the electronic whiteboard 801 launch the meeting application and enter the meeting ID of a pre-booked meeting, or select a meeting from a list of booked meetings.

[0289] S202: The conference control unit 830 of the electronic whiteboard 801 connects the electronic whiteboard 801 to the conference server 800 by sending the conference ID to the conference server 800.

[0290] S203: The conference control unit 830 of the electronic whiteboard 801 transmits its own device ID to the conference server 800. The device authentication unit 820 of the conference server 800 authenticates this device ID and allows the electronic whiteboard 801 to participate in the conference.

[0291] S204: Users of the tablet PC 802 launch the meeting application and enter the meeting ID of a pre-booked meeting, or select a meeting from a list of booked meetings.

[0292] S205: The conference control unit 840 of the tablet PC 802 sends the conference ID to the conference server 800, thereby connecting the tablet PC 802 to the conference server 800.

[0293] S206: The conference control unit 840 of the tablet PC 802 sends the user ID and password to the conference server 800. The user authentication unit 818 of the conference server 800 authenticates this user ID and password and allows the tablet PC 802 to join the conference.

[0294] S207~S209: Tablet PC 803 also participates in the meeting.

[0295] S210~S212: Next, the conference control units 830 and 840 of the electronic whiteboard 801, tablet PC 802, and tablet PC 803 establish a conference session with the conference control unit 810 of the conference server 800. A session is a communication state in which image data, audio data, document data, and whiteboard area content from the source device are distributed to all other devices participating in the same conference. S213: The audio transmission / reception unit 831 of the electronic whiteboard 801 transmits the encoded audio data input from the microphone to the conference server 800. The video transmission / reception unit 832 of the electronic whiteboard 801 transmits the encoded image data input from the camera to the conference server 800.

[0296] S214: The audio transmission / reception unit 841 of the tablet PC 802 transmits the encoded audio data input from the microphone to the conference server 800. The video transmission / reception unit 842 of the tablet PC 802 transmits the encoded image data input from the camera to the conference server 800.

[0297] S215: The tablet PC 803 also sends audio and image data to the conference server 800.

[0298] S216~S218: The conference control unit 810 of the conference server 800 transmits audio and image data received from one device to all other devices participating in the conference. The conference control unit 810 of the conference server 800 mixes the audio data received from all devices and transmits it to all other devices, while transmitting the image data received from each device to all other devices on separate channels. The audio transmission / reception units 831 and 841 of the electronic whiteboard 801, tablet PC 802, and tablet PC 803 decode the audio data received from the conference server 800 and output it as audio from their speakers. The video transmission / reception units 832 and 842 of the electronic whiteboard 801, tablet PC 802, and tablet PC 803 decode the two channels of image data received from the conference server 800 and display them in predetermined areas of the displays 480 and 526, respectively. S219: Next, the user of the electronic whiteboard 801 operates the display screen of the display 480 to turn on the whiteboard function, which is a function of the meeting application. Then, the whiteboard control unit 833 of the electronic whiteboard 801 displays the whiteboard on the display 480. The content displayed on the whiteboard at this time is the same as the content displayed in the display area 251 shown in Figure 6, but the electronic whiteboard 801 of this embodiment can also display image data from the tablet PCs 802 and 803 (see Figure 49).

[0299] S220, S221: The users of tablet PC 802 and tablet PC 803 also operate the display screen of display 526 to turn on the whiteboard function, which is a function of the meeting application. Then, the whiteboard control units 833 of tablet PC 802 and tablet PC 803 display the whiteboard area on display 526.

[0300] Figure 49 illustrates the screen displayed on the display 480 by the electronic whiteboard 801. The electronic whiteboard 801 has a whiteboard area 259 and image display areas 257 and 258 that display image data from tablet PCs 802 and 803. The whiteboard area 259 is the area that has whiteboard functions (handwriting input and drawing, etc.). The content of the display area 251 is displayed in the whiteboard area 259. Note that these sizes are examples and can be changed by the user.

[0301] Meanwhile, the display 526 of the tablet PC 802 displays image data captured by the camera of the electronic whiteboard 801 and image data of the tablet PC 803 user captured by the camera of the tablet PC 803. The display 526 of the tablet PC 803 displays image data captured by the camera of the electronic whiteboard 801 and image data of the tablet PC 802 user captured by the camera of the tablet PC 802.

[0302] <Synchronization of electronic whiteboard 801 and tablet PCs 802 and 803> Next, the user of the electronic whiteboard 801 connected their PC to the electronic whiteboard 801 using an HDMI cable (registered trademark) and displayed the PC screen data on the electronic whiteboard 801's display 480 (see Figure 7). Then, the user copied the bar graph 217 displayed on display 480 to the whiteboard. The screen of the electronic whiteboard 801 in this state looks like Figure 50.

[0303] Figure 50 illustrates the screen displayed on the display 480 by the electronic whiteboard 801. In the whiteboard area 259, the bar graph 217 displayed on the PC screen is shown as content 201.

[0304] Upon copying, the whiteboard control unit 833 of the electronic whiteboard 801 sends the display data of content 201 to the conference server 800. The conference server 800 sends this received display data to tablet PCs 802 and 803. Tablet PCs 802 and 803 display this received display data (content 201) in the whiteboard area 259.

[0305] Then, the type determination unit 813 of the conference server 800 uses a type determination model to determine that this display data is a graph (image). The content management unit 814 of the conference server 800 registers the coordinates of the top left corner of the rectangular area circumscribing this display data (content 201), the size of this rectangular area (number of display pixels in the horizontal and vertical directions), the content's additional location (location A), and the content's type (graph (image)) in the content management table in the conference data storage unit 811. The content management table can be the same as in Figure 10.

[0306] Figure 51 is a sequence diagram illustrating the synchronization process between the electronic whiteboard 801 and the tablet PCs 802 and 803.

[0307] S231: Users of each device proceed with the discussion while referring to content 201 (bar graph), but the topic of sales comes up, and the user of tablet PC 802 opens a file saved on tablet PC 802 and displays a page containing a pie chart of sales (displayed in an application other than the whiteboard). The user then wants to add this pie chart of sales to the whiteboard, but there is no free space in the whiteboard area 259, so the user of tablet PC 802 swipes within the whiteboard area to move the bar graph content 201 outside the whiteboard area of ​​display 526.

[0308] S232: When a swipe operation is performed on the tablet PC 802, the operation reception unit 846 of the tablet PC 802 receives the amount and direction of movement of the display data, and the whiteboard control unit 843 sends a display data movement command (amount of movement and direction of movement) to the conference server 800.

[0309] S233: When the conference server 800 receives this command, the whiteboard control unit 812 moves the bar graph content 201 in the display area 251 in the direction of the swipe operation. Note that the display area 251 moves in the opposite direction to the swipe operation.

[0310] Figure 52 shows content 201 and 205 placed in the display extension area 252. Content 201 was moved to the edge of the display extension area 252 by a swipe operation. Note that at the time the swipe operation was performed on the tablet PC 802, content 205 of the pie chart was not present in the display extension area 252.

[0311] S234: Next, the user of tablet PC 802 copies a pie chart of sales figures, which is displayed in an application other than the whiteboard, onto the whiteboard.

[0312] S235: Then, the whiteboard control unit 843 of the tablet PC 802 sends the display data of the pie chart content 205 for sales figures to the conference server 800.

[0313] S236: The conference server 800 places the received content 205 in the display extension area 252. The content 205 is placed in the position shown in Figure 52 in the display extension area 252 according to the previously performed swipe operation. Next, the type determination unit 813 of the conference server 800 uses a type determination model to determine that this display data is a graph (image). Then, the content management unit 814 of the conference server 800 registers the coordinates of the top left corner of the rectangular area circumscribing this display data (content 205) and the size of this rectangular area (number of display pixels in the horizontal and vertical directions) along with the content addition site (site B) and the content type (graph (image)) in the content management table in the conference data storage unit 811.

[0314] S237: Next, the user of tablet PC 802 presses the sync button to synchronize the whiteboard. The sync button displays (synchronizes) the display data of the whiteboard area of ​​the device on which the sync button was pressed to other devices participating in the meeting.

[0315] S238: As a result, the whiteboard control unit 843 of the tablet PC 802 sends a distribution request command to the conference server 800 to distribute the content of the whiteboard area of ​​the tablet PC 802 to other devices.

[0316] S239, S240: When the conference server 800 receives this command, the whiteboard control unit 812 sends the content that tablet PC 802 is displaying in the whiteboard area from the display extension area 252 to the electronic whiteboard 801 and tablet PC 803. The coordinates of the current whiteboard area of ​​tablet PC 802 are attached to the distribution request command. The content that tablet PC 802 is displaying in the whiteboard area is identified using these coordinates and the content table.

[0317] S241, S242: When the electronic whiteboard 801 and tablet PC 803 receive this content, they display it in the whiteboard area 259.

[0318] S243: Next, users of each device proceed with the discussion while referring to content 205 (a pie chart of sales figures), and at the end of the meeting, the user of the electronic whiteboard 801 swipes in the whiteboard area 259 to move content 205 outside the whiteboard area of ​​the display 480 in order to write an action item on the whiteboard.

[0319] S244: When a swipe operation is performed on the electronic whiteboard 801, the operation reception unit 836 of the electronic whiteboard 801 receives the amount and direction of movement of the displayed data, and the whiteboard control unit 833 sends a display data movement command (amount of movement and direction of movement) to the conference server 800.

[0320] S245: When the conference server 800 receives this command, the whiteboard control unit 812 moves the display area 251 of the electronic whiteboard 801 in the display extension area 252 in the opposite direction to the direction of the swipe operation.

[0321] S246: Next, the user of the electronic whiteboard 801 presses the whiteboard's sync button.

[0322] S247: The whiteboard control unit 833 sends a command to the conference server 800 requesting the distribution of the whiteboard content displayed on the display 480 of the electronic whiteboard 801.

[0323] S248, S249: When the conference server 800 receives this command, the whiteboard control unit 812 sends the content that the electronic whiteboard 801 is displaying in the whiteboard area of ​​the display extension area 252 to the tablet PCs 802 and 803. The coordinates of the current whiteboard area of ​​the electronic whiteboard 801 are attached to the distribution request command. The content that the electronic whiteboard 801 is displaying in the whiteboard area is identified using these coordinates and the content table.

[0324] S250, S251: When tablet PC 802 and tablet PC 803 receive this content, they display it in whiteboard area 259. At this point, nothing is displayed on the whiteboards of each device (only a swipe has been performed so far).

[0325] Next, the user of the electronic whiteboard 801 writes action items by hand on the whiteboard. The display extension area 252 at this time is shown in Figure 53.

[0326] Figure 53 shows an example of content placed in the display extension area 252. The display extension area 252 displays pie chart content 205 and bar chart content 201. Additionally, the display area 251 has handwritten action item content 850.

[0327] When a user of the electronic whiteboard 801 writes an action item in the whiteboard area 259, the whiteboard control unit 833 of the electronic whiteboard 801 sends this drawing data to the conference server 800. The whiteboard control unit 812 of the conference server 800 stores this drawing data in association with the electronic whiteboard 801 and also sends it to tablet PCs 802 and 803. When tablet PCs 802 and 803 receive this drawing data, they display it in the whiteboard area 259.

[0328] When drawing data is added to the display extension area 252 of the conference server 800, the type determination unit 813 uses a type determination model to determine that the added data is a drawing (hand-drawn). The content management unit 814 of the conference server 800 then registers the coordinates of the top left corner of the rectangular area circumscribing the drawing data (content 850), the size of this rectangular area (number of display pixels in the horizontal and vertical directions), the location where the content was added (location A), along with the content type (drawing (hand-drawn)) in the content management table in the conference data storage unit 811.

[0329] <Content aggregation> Next, referring to Figure 54, we will explain the process of displaying the bar graph content 201 and the pie chart content 205 on the whiteboard area 259. Figure 54 is a sequence diagram illustrating the process of placing multiple content items on the whiteboard area 259.

[0330] S261: When the user of the electronic whiteboard 801 attempted to handwrite the content 850 of the action item, it became necessary to check the bar graph content 201 and the sales revenue pie chart content 205. Therefore, the user of the electronic whiteboard 801 pressed the aggregate display menu button 253 shown in Figure 49.

[0331] S262: The display control unit 837 of the electronic whiteboard 801 then displays the condition setting screen 260 shown in Figure 11. Here, the user selects "Graph (Image)" from the content types.

[0332] S263: The whiteboard control unit 833 of the electronic whiteboard 801 sends a command to the conference server 800 requesting content aggregation, including graph (image) information as the content type.

[0333] S264: When the conference server 800 receives this command, the aggregation processing unit 815 aggregates the content of the bar graph 201 and the content of the pie chart 205, which are the contents of the graph (image), into the display area 251. However, since the two contents 201 and 205 do not fit within the display area 251, the aggregation processing unit 815 shrinks the contents 201 and 205 so that they fit within the display area 251.

[0334] S265~S267: The whiteboard control unit 812 then transmits the content 201 and 205 within the display area 251 to the electronic whiteboard 801, tablet PC 802, and tablet PC 803.

[0335] S268~S270: When the electronic whiteboard 801, tablet PC 802, and tablet PC 803 receive this content 201 and 205, the display control units 837 and 847 display it in the whiteboard area 259.

[0336] In this way, when the user of the electronic whiteboard 801 performs the aggregate display operation, the specified type of content is aggregated and displayed on the displays of all devices participating in the meeting.

[0337] Figure 55 is an example flowchart illustrating the operation of the conference server 800.

[0338] The conference server 800 determines whether or not it has received the content (S281). If the determination in step S281 is Yes, the process proceeds to step S282; otherwise, the process proceeds to step S284.

[0339] In step S282, the conference server 800's type determination unit 813 determines the type of display data (S282). Then, the conference server 800's content management unit 814 registers the size of the rectangular area surrounding the display data, the coordinates of the top left corner, the additional location, and the content type in the content management table (S283).

[0340] In step S284, it is determined whether the whiteboard control unit 812 of the conference server 800 has received a display data transfer command (S284). If the determination in step S284 is Yes, the process proceeds to step S285; otherwise, the process proceeds to step S286.

[0341] In step S285, the whiteboard control unit 812 of the conference server 800 moves the content in the display extension area 252 (S285).

[0342] In step S286, the whiteboard control unit 812 of the conference server 800 determines whether or not it has received a command to request the distribution of content in the whiteboard area (S286). If the determination in step S286 is Yes, the process proceeds to step S287; otherwise, the process proceeds to step S288.

[0343] In step S287, the whiteboard control unit 812 of the conference server 800 transmits the content of the whiteboard area 259 of the requesting device to all other devices except the requesting device (S287).

[0344] In step S288, the whiteboard control unit 812 of the conference server 800 determines whether or not it has received a content aggregation request command (S288). If the determination in step S288 is Yes, the process proceeds to step S289; otherwise, the process proceeds to step S281.

[0345] In step S289, the aggregation processing unit 815 of the conference server 800 aggregates the specified type of content into the display area 251 (S289).

[0346] Then, the whiteboard control unit 812 of the conference server 800 transmits the aggregated display data to all devices (S290).

[0347] <Main effects> In this embodiment, the display system allows users of each device to input content and, when they press the sync button, display that content on other devices.

[0348] [Fourth Embodiment] In the third embodiment, when the user wanted to display the content of the whiteboard area 259, which was being displayed on one device, on another device, the user pressed the whiteboard's sync button. As a result, the device sent a command to the conference server 800 requesting the distribution of the whiteboard area content, and the conference server 800 sent the content display data to the other device.

[0349] In this embodiment, we will describe a case in which the content of the whiteboard area 259 displayed on another device is displayed on the user's device at the user's discretion, without the user of that device having to operate the other device.

[0350] The system configuration, hardware configuration, and functional configuration of the conference server 800, electronic whiteboard 801, tablet PC 802, and tablet PC 803 are the same as in the third embodiment.

[0351] The electronic whiteboard 801 and tablet PCs 802 and 803 transmit and receive audio data and image data from the camera input. Furthermore, when the user of the electronic whiteboard 801 presses the whiteboard synchronization button, the bar graph content 201 shown in Figure 50 is displayed on the other devices as well.

[0352] Figure 56 is a sequence diagram illustrating the process of sending the edit flag.

[0353] Users of each device are discussing while referring to content 201, but the user of tablet PC 802 moves the bar graph content 201 by swiping in order to input handwriting.

[0354] S301, S302: When a swipe operation is performed on the tablet PC 802, the operation reception unit 846 of the tablet PC 802 sends a display data movement command to the conference server 800, including the amount and direction of movement of the content 201.

[0355] S303: When the conference server 800 receives this command, the whiteboard control unit 812 moves the display area of ​​the tablet PC 802 in the display extension area 252 in the opposite direction to the swipe operation.

[0356] S304: The user handwrites relevant information in the available space. Here, the user is adding new content, but they may also edit existing content.

[0357] S305: In addition, the whiteboard control unit 843 of the tablet PC 802 sends handwritten drawing data to the conference server 800.

[0358] S306: The whiteboard control unit 812 of the conference server 800 stores the content of this drawing data in association with the tablet PC 802. That is, the type determination unit 813 of the conference server 800 uses a type determination model to determine that this display data is a drawing (handwritten). Subsequently, the content management unit 814 of the conference server 800 registers the coordinates of the top left corner of the rectangular area circumscribing this display data, the size of the rectangular area (number of display pixels in the horizontal and vertical directions), and the content addition location (location B) along with the content type (drawing (handwritten)) in the content management table in the conference data storage unit 811.

[0359] S307, S308: In order to display on other devices that handwriting input is being accepted, the whiteboard control unit 812 of the conference server 800 then sends information (editing flag = on) to the electronic whiteboard 801 and tablet PC 803 indicating that handwriting has been drawn on the whiteboard by the tablet PC 802. S309: When the electronic whiteboard 801 receives this information, the display control unit 837 displays "Editing" below the image display area 257 of the tablet PC 802.

[0360] Figure 57 illustrates an example of what the electronic whiteboard 801 displays on the display 480. Figure 57 primarily explains the differences from Figure 50. In Figure 57, the image data from the tablet PC 802 displays "Editing" 271.

[0361] S310: When the tablet PC 803 also receives this information, the display control unit 847 displays "Editing" below the image data from the tablet PC 802.

[0362] S311: Next, the user of electronic whiteboard 801 presses "Editing" 271.

[0363] S312: As a result, the whiteboard control unit 833 sends a command to the conference server 800 to retrieve the entered or currently entered content displayed in the whiteboard area 259 of the tablet PC 802.

[0364] S313: When the conference server 800 receives this command, the whiteboard control unit 812 sends the entered or currently entered content associated with the whiteboard area 259 of the tablet PC 802 to the electronic whiteboard 801.

[0365] S314: The display control unit 837 of the electronic whiteboard 801 displays the received content in the whiteboard area 259. Figure 58 shows the content 206 that the electronic whiteboard 801 displays in the whiteboard area 259.

[0366] Figure 58 illustrates the content 206 displayed on the display 480 by the electronic whiteboard 801. Compared to Figure 57, the content 206 displayed is handwritten drawing data by the user of the tablet PC 802 (either completed or in progress). The whiteboard area 259 of the electronic whiteboard 801 displays the content 201 and 206 of the whiteboard area of ​​the tablet PC 802.

[0367] In Figure 58, the content 206 of the tablet PC 802 is displayed by overwriting the whiteboard area 259 of the electronic whiteboard 801, but as shown in Figure 59, the whiteboard area 259 of the electronic whiteboard 801 can remain displayed as is.

[0368] Figure 59 illustrates a thumbnail 272, which is a reduced-size representation of the whiteboard area 259 of the tablet PC 802. The thumbnail 272 is a reduced-size image of the whiteboard area 259 of the tablet PC 802. An "Enlarge" icon 273 (an example of a display component) is displayed in the upper right corner of the thumbnail 272. When the user presses this "Enlarge" icon 273, the electronic whiteboard 801 displays the contents of the tablet PC 802's whiteboard on the electronic whiteboard 801's whiteboard at 1:1 scale, as shown in Figure 58. Note that the thumbnail may be displayed not only when the user of the electronic whiteboard 801 presses "Editing" 271, but also when the pointer is hovered over "Editing" 271. Hereafter, the operation of "pressing" or "hovering the pointer" will be referred to as "display selection".

[0369] Figure 60 is a sequence diagram illustrating the process when the edit flag is cleared.

[0370] S321: The whiteboard control unit 812 of the conference server 800 determines that there is no handwriting if it stops receiving handwritten drawing data from the tablet PC 802 for a predetermined period of time.

[0371] S322, S323: The whiteboard control unit 812 of the conference server 800 sends information (editing flag = off) to the electronic whiteboard 801 and tablet PC 803 indicating that handwritten drawing on the whiteboard has been completed on the tablet PC 802.

[0372] S324: When the electronic whiteboard 801 receives this information, the display control unit 837 erases the "Editing" 271 that is displayed below the image data from the tablet PC 802.

[0373] S325: When the tablet PC 803 also receives this information, the display control unit 847 erases the "Editing" message displayed below the image data from the tablet PC 802.

[0374] S326: Next, the user of tablet PC 803 moved the whiteboard area 259 over the pie chart content 205 using a swipe gesture and wrote related information by hand in the empty space.

[0375] S327: The whiteboard control unit 843 of the tablet PC 803 sends this handwritten drawing data to the conference server 800.

[0376] S328: The whiteboard control unit 812 of the conference server 800 stores this drawing data in association with the tablet PC 803. That is, the type determination unit 813 of the conference server 800 uses a type determination model to determine that this display data is a drawing (handwritten). Subsequently, the content management unit 814 of the conference server 800 registers the coordinates of the top left corner of the rectangular area circumscribing this display data, the size of this rectangular area (number of display pixels in the horizontal and vertical directions), and C as the content addition point, along with the content type (drawing (handwritten)), in the content management table in the conference data storage unit 811.

[0377] S329, S330: The whiteboard control unit 812 of the conference server 800 sends information (editing flag = on) to the electronic whiteboard 801 and tablet PC 802 indicating that handwritten drawings have been made on the whiteboard by the tablet PC 803.

[0378] S331: When the electronic whiteboard 801 receives this information, the display control unit 837 displays "Editing" 274 below the image display area 258 of the tablet PC 803.

[0379] S332: When the tablet PC 802 also receives this information, the display control unit 847 displays "Editing" below the image display area of ​​the tablet PC 803.

[0380] Figure 61 shows an example of content placement within the display extension area 252. Content 207, which is drawing data added by the user of tablet PC 803, is displayed. Content 206, on the other hand, was handwritten by the user of tablet PC 802, as shown in Figure 58.

[0381] S333: Next, the user of the electronic whiteboard 801 presses the "Editing" button 274 displayed below the image display area 258 on the tablet PC 803.

[0382] S334: The whiteboard control unit 833 sends a command to the conference server 800 to retrieve the content of the whiteboard area 259 on the tablet PC 803.

[0383] S335: When the conference server 800 receives this command, the whiteboard control unit 812 sends the content associated with the whiteboard area 259 of the tablet PC 803 to the electronic whiteboard 801.

[0384] S336: The display control unit 837 of the electronic whiteboard 801 displays the received content.

[0385] Figure 62 shows the content that the electronic whiteboard 801 displays on the whiteboard area 259. In Figure 62, the electronic whiteboard 801 displays content 205 and 207, which are acquired from the tablet PC 803 and displayed on the whiteboard area 259 by the tablet PC 803. Therefore, the electronic whiteboard 801 and the tablet PC 803 are synchronized. In addition, "Editing" 274 is displayed below the image display area 258 where the image data from the tablet PC 803 is displayed. Alternatively, instead of displaying the content acquired from the tablet PC 803 across the entire whiteboard area 259 as shown in Figure 62, the content acquired from the tablet PC 803 may be displayed as thumbnails. Specifically, content 201 may be displayed in the whiteboard area 259, while content 205 and 207, which are outside the whiteboard area 259, are displayed only as thumbnails. The thumbnail may be displayed not only when the user of the electronic whiteboard 801 presses "Editing" 271, but also when the pointer is hovered over "Editing" 274 (when the display of "Editing" 274 is selected).

[0386] At this point, the user of tablet PC 803 is not viewing content 206 of the drawing data for Figure 58, which was handwritten by the user of tablet PC 802. Also, the user of tablet PC 802 is not viewing content 207 of the drawing data for Figure 62, which was handwritten by the user of tablet PC 803. Therefore, the user of electronic whiteboard 801 decides to aggregate the handwritten information from both and send it to tablet PC 802 and tablet PC 803. The user of electronic whiteboard 801 presses the aggregate display menu button 253. This displays the condition setting screen. The user can also set the system to display the image of the whiteboard area 259 of electronic whiteboard 801 as is, in which case the thumbnails of content 205 and 207, which are displayed when "Editing" is selected, will also be displayed to each user (without making a decision on whether to send them).

[0387] Figure 63 shows the condition setting screen 260 displayed by the electronic whiteboard 801. The keyword field 261 is an input field for keywords used to search for content. Users can enter requests, questions, or keywords in the keyword field 261. • The type specification field 262 displays the type of content to be displayed using a pull-down menu or similar. • Participant field 265 is a field for specifying a participant and displaying the handwritten drawing data of that participant. The aggregate button 263 is a button used to aggregate the content of the display extension area 252.

[0388] The condition setting screen 260 in Figure 63 has a participant field 265 added to the condition setting screen 260 shown in Figure 11.

[0389] Figure 64 is a sequence diagram illustrating the process of sending content to the electronic whiteboard 801, tablet PC 803, and tablet PC 802.

[0390] S341: The user of electronic whiteboard 801 presses the aggregate display menu button 253.

[0391] S342: The electronic whiteboard 801 displays the condition setting screen 260 shown in Figure 63, and the conference control unit 830 sends a participant list acquisition command to the conference server 800.

[0392] S343: When the conference server 800 receives this command, the conference control unit 810 retrieves the names of the participants in this conference from the conference data storage unit 811 and sends this participant list back to the electronic whiteboard 801.

[0393] S344: The electronic whiteboard 801 receives the participant list. When a user of the electronic whiteboard 801 touches the participant field, the display control unit 837 displays a list of participant names received from the conference server 800 in a pull-down menu in the participant field 265.

[0394] S345: The user of electronic whiteboard 801 selects the user of tablet PC 802 and the user of tablet PC 803, selects drawing (handwriting) as the content type, and presses the aggregate button 263.

[0395] S346: The whiteboard control unit 833 of the electronic whiteboard 801 sends a command to the conference server 800 requesting the aggregation of content, including this selection information.

[0396] S347: When the conference server 800 receives this command, the aggregation processing unit 815 refers to the content management table in the conference data storage unit 811 and gathers the content of the drawing (handwritten) at location B and the content of the drawing (handwritten) at location C into the display area 251. The conference control unit 810 of the conference server 800 stores participant names and their respective locations in the conference data storage unit 811.

[0397] S348~S350: The whiteboard control unit 812 of the conference server 800 transmits the content in this display area 251 to the electronic whiteboard 801, tablet PC 802, and tablet PC 803.

[0398] S351~S353: When the display control units 837 and 847 of the electronic whiteboard 801, tablet PC 802, and tablet PC 803 receive this content, they display it in the whiteboard area 259.

[0399] Figure 65 shows the content that the interactive whiteboard 801 displays on the whiteboard area 259. The same screen as in Figure 65 is also displayed on the tablet PCs 802 and 803. In Figure 65, the content is arranged in order of largest area. Content 206 was entered by the user of tablet PC 802 using handwriting input, and drawing data content 207 was entered by the user of tablet PC 803 using handwriting input. In this way, the interactive whiteboard 801 can display aggregated content based on the type of content selected by the user and the content associated with that user.

[0400] <Main effects> In this embodiment, the display system allows users on other devices to display content when they input content on one device and then press the "Editing" button. It can aggregate and display content associated with a user based on the type of content selected by that user.

[0401] [Fifth Embodiment] In this embodiment, we will describe a case in which the electronic whiteboard 801 instructs the display of aggregated information to be provided by voice rather than by operating a menu displayed on the screen.

[0402] The system configuration and the hardware configuration of the conference server 800, electronic whiteboard 801, tablet PC 802, and tablet PC 803 are the same as in the third embodiment.

[0403] Figure 66 shows a functional configuration diagram of the conference server 800 of this embodiment. The explanation of Figure 66 mainly describes the differences from Figure 45. The conference server 800 in Figure 66 newly includes a speech recognition unit 860, a speech text generation unit 861, and an instruction content determination unit 862. The other functional units are the same as those shown in Figure 45.

[0404] The speech recognition unit 860 decodes encoded speech data received from other devices and outputs text corresponding to the speech data through speech recognition.

[0405] The speech text generation unit 861 generates text based on the text data output from the speech recognition unit 860.

[0406] The instruction content determination unit 862 determines the instruction content from the text generated by the utterance text generation unit 861. The instruction content determination unit 862 improves the accuracy of instruction content determination by using an instruction tuning method that trains the unit on specific instructions. Instruction tuning is a method for fine-tuning a large-scale language model (LLM) using a labeled dataset of instruction content and its corresponding output.

[0407] Figure 67 shows an example of the correspondence between instruction sentences learned through instruction tuning and their corresponding outputs (tasks). Therefore, once the instruction sentence is clarified, the task is identified.

[0408] First, similar to the third embodiment, a remote meeting is conducted between the electronic whiteboard 801, tablet PC 802, and tablet PC 803 using audio data, image data, and a whiteboard. Content as shown in Figure 1 is placed in the display extension area 252. In this state, when the user of the electronic whiteboard 801 says "Please display only the graph," the audio transmission / reception unit 831 of the electronic whiteboard 801 encodes the audio data, which is A / D converted from the audio signal input from the microphone 440, using the AAC method and sends it to the conference server 800. The conference server 800 decodes this audio data, the speech recognition unit 860 outputs text corresponding to the audio data through speech recognition, and the speech text generation unit 861 generates text based on this text data.

[0409] Next, the instruction content determination unit 862 determines the instruction content (task) from the text generated by the utterance text generation unit 861. Based on the correspondence in Figure 67, the instruction content determination unit 862 identifies the task of aggregated display of graphs (images).

[0410] Next, the aggregation processing unit 815 refers to the content management table in the conference data storage unit 811 and gathers the bar graph (content 201) and pie chart (content 205) into the display area 251. Then, the whiteboard control unit 812 of the conference server 800 transmits the data in this display area 251 to the electronic whiteboard 801, tablet PC 802, and tablet PC 803. Upon receiving this data, the electronic whiteboard 801, tablet PC 802, and tablet PC 803 display it in the whiteboard area 259. An example of the display on the electronic whiteboard 801 at this time is the same as in Figure 19.

[0411] Figure 68 is a flowchart illustrating the processing of the conference server 800, which receives instructions for aggregated display via voice.

[0412] The communication control unit 817 of the conference server 800 determines whether or not it has received audio data (S501).

[0413] If audio data is received (Yes in S501), the speech recognition unit 860 of the conference server 800 converts the audio data into text using speech recognition and generates a document from this text (S502).

[0414] Next, the instruction content determination unit 862 of the conference server 800 determines the instruction content (task) from the text by referring to the correspondence in Figure 67 (S503).

[0415] The instruction content determination unit 862 of the conference server 800 determines whether the task is to display a summary of graphs (images) (S504). If the determination in step S504 is Yes, the process proceeds to step S505; if No, the process proceeds to step S506.

[0416] In step S505, the content management unit 814 of the conference server 800 extracts the content of the graph (image) and aggregates the content according to the arrangement rules (S505).

[0417] In step S506, the instruction content determination unit 862 of the conference server 800 determines whether it is another task (S506). If it is another task, the conference server 800 executes the processing corresponding to the task (S507). This returns the process to step S501.

[0418] Next, the whiteboard control unit 812 of the conference server 800 determines whether the aggregated content extends beyond the display area 251 (S508). If the determination in step S508 is Yes, the process proceeds to step S509; otherwise, the process proceeds to step S513.

[0419] In step S509, the aggregation processing unit 815 of the conference server 800 determines whether the overhang is in only one direction, either the X-axis or Y-axis (S509). If the determination in step S509 is Yes, the process proceeds to step S510; otherwise, the process proceeds to step S511.

[0420] In step S510, the aggregation processing unit 815 of the conference server 800 calculates the reduction ratio in the direction of overflow (S510).

[0421] In step S511, the aggregation processing unit 815 of the conference server 800 calculates the reduction ratio in both the X-axis and Y-axis directions, and sets the smaller value as the reduction ratio (S511).

[0422] The aggregation processing unit 815 of the conference server 800 displays the entire aggregated content at a reduced size (S512).

[0423] In step S513, the whiteboard control unit 812 of the conference server 800 displays the entire aggregated content as is (S513).

[0424] <Main effects> The display system of this embodiment can consolidate multiple pieces of content using voice.

[0425] [Other application examples] Although the best mode for carrying out the present invention has been described above using examples, the present invention is not limited in any way to these examples, and various modifications and substitutions can be made without departing from the spirit of the present invention.

[0426] For example, in this embodiment, the user wrote by hand on an electronic whiteboard, but text input using a keyboard or voice control may also be used. Scrolling may also be performed using voice control.

[0427] Alternatively, the electronic whiteboard 2 of the first embodiment and the VR goggles may communicate, and the VR goggles may display the content aggregated in the display area 251.

[0428] Furthermore, in this embodiment, the user could determine the scroll amount of the display area through operation, but the scroll amount may be fixed in page units (one screen). Alternatively, the electronic whiteboard 2 may automatically determine the minimum scroll amount required to make the display area 251 blank and automatically scroll by this amount. In this case, when the user performs a scroll operation, the display area 251 will scroll by the minimum scroll amount required to make it blank, regardless of the actual scroll amount.

[0429] Furthermore, an electronic blackboard may also be called an electronic whiteboard, electronic information board, etc. Moreover, this embodiment is not limited to electronic blackboards; it can be suitably applied to any information processing device having a touch panel. Examples of information processing devices equipped with touch panels include PCs, tablet terminals, and smartphones. While these are normally general-purpose information processing devices, when an application that functions as a display device is executed, the user can operate them as a display device.

[0430] Furthermore, the configuration examples shown in Figure 5 and other figures are divided according to their main functions to facilitate understanding of the processing performed by the electronic whiteboard 2. The present invention is not limited by the way the processing units are divided or the names of those units. The processing of the electronic whiteboard 2 can be further divided into many more processing units depending on the processing content. It is also possible to divide the processing units so that each unit includes even more processing.

[0431] In this embodiment, the contact coordinates were detected by detecting the pen tip coordinates on a touch panel, but the electronic whiteboard 2 may also detect the pen tip coordinates using ultrasound. The pen emits ultrasound along with light emission, and the electronic whiteboard 2 calculates the distance based on the arrival time of the ultrasound. The electronic whiteboard 2 can determine the position of the pen based on its direction and distance. The electronic whiteboard 2 displays the drawing data on its display. In this method, the electronic whiteboard 2 can also use a projector instead of a display, and project the pen's trajectory as drawing data onto the projector.

[0432] Furthermore, each function of the embodiments described above can be realized by one or more processing circuits. Hereinafter, "processing circuit" as used herein includes processors programmed to execute each function by software, such as processors implemented by electronic circuits, as well as devices such as ASICs (Application Specific Integrated Circuits), DSPs (digital signal processors), FPGAs (field programmable gate arrays), and conventional circuit modules designed to execute each of the functions described above.

[0433] Embodiments of the present invention bring about significant improvements in the capabilities and functionality of computers. These improvements enable users to utilize computers that provide more efficient and robust interaction with tables, which are methods for storing and presenting information in an information processing device. Furthermore, embodiments of the present invention provide a better user experience through the use of a more efficient, powerful, and robust user interface. Such a user interface provides better interaction between humans and machines.

[0434] <Nature> [Aspect 1] An information processing device that can scroll the display area to display the content entered in the display extension area, A process of extracting one or more content items from the content entered in the display extension area based on the type of content, A process of aggregating one or more extracted contents into the display area, A method of displaying the information. [Aspect 2] The aforementioned information processing device is The process involves receiving a selection of the content type from the user. The received content of the aforementioned type is extracted from the content entered in the display extension area. The extracted content is aggregated in the display area. The display method described in Embodiment 1. [Aspect 3] The aforementioned information processing device is Only the types of content present in the aforementioned display extension area are displayed as a list. The process involves accepting the selection of a content type from the aforementioned list of content types. The display method described in Embodiment 2. [Aspect 4] The aforementioned information processing device is If one or more of the contents aggregated in the display area do not fit within the display area, The process involves reducing the size of one or more of the content aggregated in the display area so that it fits within the display area. The display method described in any one of the three descriptions (1 to 3). [Aspect 5] One or more of the aforementioned contents are Location information corresponding to the position of each content based on a predetermined origin, It includes size information corresponding to the size of each piece of content, The aforementioned information processing device is The display method according to embodiment 2, characterized in that the content of the type specified by the user is displayed in such a way that it fits within the display area, using the position information and the size information. [Aspect 6] The display method according to embodiment 5, characterized in that the position information is information indicating a predetermined end of each of the contents. [Aspect 7] The display method according to embodiment 6, characterized in that the size information is calculated based on information indicating a predetermined end of each content and information indicating the other end of each content. [Aspect 8] The display method according to embodiment 7, characterized in that the size information is information calculated from the difference between information indicating a predetermined end and information indicating another end. [Aspect 9] The size information includes information indicating a predetermined end, information indicating another end, and information indicating another end different from the other end. The display method according to embodiment 8, characterized in that it has a second size information calculated based on the information indicating the predetermined end and the information indicating another other end. [Aspect 10] The display method according to embodiment 9, characterized in that the second size information is information calculated from the difference between information indicating a predetermined end and information indicating another other end. [Aspect 11] The display method according to embodiment 10, characterized in that each of the contents is arranged in accordance with a predetermined rule and displayed in the display area. [Aspect 12] The display method according to embodiment 11, characterized in that the predetermined rule is to resize the content based on the size information of each content and display it in the display area. [Aspect 13] The information processing device includes an object extraction unit that groups the content of the aforementioned type received from the user, The display method according to embodiment 2, characterized in that the content of the type received from the user is grouped by the object extraction unit. [Aspect 14] The aforementioned information processing device is When the content is entered into the display area, A process for determining the type of input content, The process involves registering the determined type in the content management table, associating it with the aforementioned content. The display method described in any one of the three manifestations 1 to 13. [Aspect 15] The aforementioned information processing device is The system accepts keyword input from the user. The received keywords are extracted from the content entered in the display extension area, The extracted content is aggregated in the display area. The display method described in Embodiment 3. [Aspect 16] The aforementioned information processing device can communicate with multiple devices, One or more content items aggregated in the aforementioned display area are transmitted to the aforementioned multiple devices for display. The display method described in Embodiment 1. [Aspect 17] The information processing device holds identification information of the user who added or edited the content in association with the content. When the selection of the type of content and the selection of the user's identification information who added or edited the content are accepted, The content associated with the user and of the type received is extracted from the content entered in the display extension area. The extracted content is aggregated in the display area. The display method described in Embodiment 1. [Aspect 18] When the information processing device detects a device that is receiving input to the display extension area, The fact that the aforementioned input is being received is displayed on other devices, associated with the device that is receiving the aforementioned input. The display method described in Embodiment 16. [Aspect 19] The aforementioned input is characterized by being handwritten input. The display method described in Embodiment 18. [Aspect 20] The aforementioned information processing device receives image data of the users of the multiple devices from the devices, The other device is made to display a message indicating that the aforementioned input has been received, in association with the image data transmitted from the device that has received the aforementioned input. The display method described in aspect 18 or 19. [Aspect 21] The device has accepted the selection of a display indicating that it has accepted the aforementioned input, When the information processing device receives a request from another device for content displayed in the whiteboard area of ​​the device that is receiving the input, it transmits the display data of the whiteboard area of ​​the device that is receiving the input to the other device that received the display selection. The aforementioned other device displays the display data in a reduced size and also displays a display component that accepts enlargement. When the aforementioned display component is pressed, the display data is enlarged and displayed. The display method described in aspect 18 or 19. [Aspect 22] The device has accepted the option to display that it is accepting input, The inputted or currently inputting content of the device receiving the input is transmitted to the other device that received the display selection. The aforementioned other device displays the content that has been entered or is currently being entered. The display method described in aspect 18 or 19. [Aspect 23] The device has accepted the option to display that it is accepting handwritten input. When the information processing device receives a request from another device for content displayed in the whiteboard area of ​​the device that is receiving the input, it transmits the display data of the whiteboard area of ​​the device that is receiving the input to the other device that has received the display selection. The other devices mentioned above display the display data. The display method described in aspect 18 or 19. [Aspect 24] An information provision method that displays content using the display method of Embodiment 1. [Pattern 25] The information provision method according to aspect 24, which causes one or more of the aforementioned contents extracted based on a request, question, or keyword entered by a user into the device to be displayed on a device that can communicate with the information processing device. [Aspect 26] The information provision method according to embodiment 25, characterized in that, after performing a process to aggregate one or more contents extracted based on the aforementioned request, question, or keyword into the display area, the contents are displayed on a device that can communicate with the information processing device. [Aspect 27] The information provision method according to embodiment 26, characterized in that it performs a process to display on a device that can communicate with the information processing device such that the proportion of content related to the request, question, or keyword that occupies the display area increases. [Aspect 28] The information provision method according to embodiment 24, characterized by performing a process to aggregate and display content related to the aforementioned request, question, or keyword on both sides or in the center of the display area. [Aspect 29] The information provision method according to embodiment 25, characterized in that the aforementioned request, question, or keyword is input by voice. [Explanation of Symbols]

[0435] 2, 2A, 2B, Electronic whiteboard 3. Tablet PC 5. Virtual meeting server 10, 12 VR goggles 11, 13 VR control controller 17, 19 PC 300 Display Systems 700 Virtual Meeting Systems [Prior art documents] [Patent Documents]

[0436] [Patent Document 1] Japanese Patent Publication No. 2016-170676< / pc>

Claims

1. An information processing device that can scroll the display area to display the content entered in the display extension area, A process of extracting one or more content items from the content entered in the display extension area based on the type of content, A process of aggregating one or more extracted contents into the display area, A method of displaying the information.

2. The aforementioned information processing device is The process involves receiving a selection of the content type from the user. The received content of the aforementioned type is extracted from the content entered in the display extension area. The extracted content is aggregated in the display area. The display method according to claim 1.

3. The aforementioned information processing device is Only the types of content present in the aforementioned display extension area are displayed as a list. The process involves accepting the selection of a content type from the aforementioned list of content types. The display method according to claim 2.

4. The aforementioned information processing device is If one or more of the contents aggregated in the display area do not fit within the display area, The process involves reducing the size of one or more of the content aggregated in the display area so that they fit within the display area. The display method according to any one of claims 1 to 3.

5. One or more of the aforementioned contents are Location information corresponding to the position of each content based on a predetermined origin, It includes size information corresponding to the size of each piece of content, The aforementioned information processing device is The display method according to claim 2, characterized in that the content of the type specified by the user is displayed in such a way that it fits within the display area, using the position information and the size information.

6. The display method according to claim 5, characterized in that the position information is information indicating a predetermined end of each of the contents.

7. The display method according to claim 6, characterized in that the size information is calculated based on information indicating a predetermined end of each content and information indicating the other end of each content.

8. The display method according to claim 7, characterized in that the size information is information calculated from the difference between information indicating a predetermined end and information indicating another end.

9. The size information includes information indicating a predetermined end, information indicating other ends, and information indicating an end other than the predetermined end and the other ends. The display method according to claim 8, characterized in that it has a second size information calculated based on the information indicating the predetermined end and the information indicating the other end.

10. The display method according to claim 9, characterized in that the second size information is information calculated from the difference between information indicating a predetermined end and information indicating another end.

11. The display method according to claim 10, characterized in that each of the aforementioned contents is arranged in accordance with a predetermined rule and displayed in the display area.

12. The display method according to claim 11, characterized in that the predetermined rule is to resize the content based on the size information of each content and display it in the display area.

13. The information processing device includes an object extraction unit that groups the content of the aforementioned type received from the user, The display method according to claim 2, characterized in that the content of the type received from the user is grouped by the object extraction unit.

14. The aforementioned information processing device is When the content is entered into the display area, A process for determining the type of input content, The process involves registering the determined type in the content management table, associating it with the aforementioned content. The display method according to claim 1, which performs the following.

15. The aforementioned information processing device is The system accepts keyword input from the user. The received keywords are extracted from the content entered in the display extension area, The extracted content is aggregated in the display area. The display method according to claim 3.

16. The aforementioned information processing device can communicate with multiple devices, One or more content items aggregated in the aforementioned display area are transmitted to the aforementioned multiple devices for display. The display method according to claim 1.

17. The information processing device holds identification information of the user who added or edited the content in association with the content. When the selection of the type of content and the selection of the user's identification information who added or edited the content are accepted, The content associated with the user and of the type received is extracted from the content entered in the display extension area. The extracted content is aggregated in the display area. The display method according to claim 1.

18. When the information processing device detects a device that is receiving input to the display extension area, The fact that the aforementioned input is being received is displayed on other devices, associated with the device that is receiving the aforementioned input. The display method according to claim 16.

19. The aforementioned input is characterized by being handwritten input. The display method according to claim 18.

20. The aforementioned information processing device receives image data of the users of the multiple devices from the devices, The other device is made to display a message indicating that the aforementioned input has been received, in association with the image data transmitted from the device that has received the aforementioned input. The display method according to claim 18 or 19.

21. The device has accepted the selection of a display indicating that it has accepted the aforementioned input, When the information processing device receives a request from another device for content displayed in the whiteboard area of ​​the device that is receiving the input, it transmits the display data of the whiteboard area of ​​the device that is receiving the input to the other device that received the display selection. The aforementioned other device displays the display data in a reduced size and also displays a display component that accepts enlargement. When the aforementioned display component is pressed, the display data is enlarged and displayed. The display method according to claim 18 or 19.

22. The device has accepted the option to display that it is accepting input, The inputted or currently inputting content of the device receiving the input is transmitted to the other device that received the display selection. The aforementioned other device displays the content that has been entered or is currently being entered. The display method according to claim 18 or 19.

23. The device has accepted the option to display that it is accepting handwritten input. When the information processing device receives a request from another device for content displayed in the whiteboard area of ​​the device that is receiving the input, it transmits the display data of the whiteboard area of ​​the device that is receiving the input to the other device that has received the display selection. The other devices mentioned above display the display data. The display method according to claim 18 or 19.

24. A method for providing information that displays content using the display method of claim 1.

25. The information provision method according to claim 24, wherein the information processing device causes the device to display one or more of the contents extracted based on a request, question, or keyword entered by a user into the device.

26. The information provision method according to claim 25, characterized in that, after performing a process to aggregate one or more contents extracted based on the aforementioned requests, questions, or keywords into the display area, the information processing device displays them on the device.

27. The information provision method according to claim 26, characterized in that the information processing device performs processing to display on the device such that the proportion of the display area occupied by content related to the request, question, or keyword increases.

28. The information provision method according to 26, characterized in that the information processing device performs a process of aggregating and displaying content related to the request, question, or keyword on both sides or in the center of the display area.

29. The information provision method according to claim 25, characterized in that the aforementioned request, question, or keyword is input by voice.

30. A display system in which a first information processing device and a second information processing device can communicate via a network, and the first information processing device can scroll the display area to display the content entered in the display extension area, The aforementioned second information processing device is The display area has a virtual whiteboard synchronized with the display area, and a virtual space where the content moved to the display extension area is placed. A process of extracting one or more pieces of content from the content placed in the virtual space based on the type of content, The process involves aggregating one or more of the extracted content onto the virtual whiteboard, The first information processing device is The process of displaying one or more of the contents aggregated on the virtual whiteboard, which have been received from the second information processing device, Display system.

31. The second information processing device can communicate with a terminal device via a network. The process involves placing the screen data transmitted from the terminal device into the virtual space and copying the screen data to the virtual whiteboard. A process of sharing the screen data placed on the virtual whiteboard with the first information processing device, The display system according to claim 30, which performs the following:

32. The aforementioned second information processing device is The first information processing device receives swipe information relating to an operation in which the screen data displayed as content is swept to the display extension area. The screen data copied to the virtual whiteboard is moved to the virtual space other than the virtual whiteboard according to the swipe information, and the virtual whiteboard is made blank. The display system according to claim 31.

33. A display system in which a first information processing device and a second information processing device can communicate via a network, and the first information processing device can scroll the display area to display the content entered in the display extension area, The first information processing device is The content receiving unit receives the content input to the display extension area from the second information processing device, An aggregation processing unit that extracts one or more of the content, including the received content, from the content input to the display extension area based on the type of content, and aggregates the extracted one or more of the content into the display area, The aggregation processing unit includes a display control unit that causes one or more of the aggregated contents to be displayed in the display area, A display system having the following features.

34. An information processing device that can scroll the display area to display the content entered in the display extension area, A consolidation processing unit that extracts one or more content items from the content input to the display extension area based on the type of content, and aggregates the extracted one or more content items into the display area, The aggregation processing unit includes a display control unit that causes one or more of the aggregated contents to be displayed in the display area, An information processing device having

35. A head-mounted display that can communicate via a network with the second information processing device in the display system according to any one of claims 30 to 33, The system receives one or more of the aforementioned contents aggregated in the display area and displays them on the display unit. Head-mounted display.

36. One or more of the contents aggregated in the display area are: The head-mounted display according to claim 35, which is displayed so as to fit within the display area based on size information calculated from information indicating a predetermined end of one or more of the contents and information indicating other ends of one or more of the contents.

37. The types of content present in the aforementioned display extension area are displayed as a list. The head-mounted display according to claim 35, which accepts the selection of the type of content from the user.

Images