Information processing device and information display method

Abstract

To display drawing data of each base while restraining the drawing data of each base from becoming small.SOLUTION: An information processing device capable of displaying any display area of a drawable area or communicating with a plurality of devices for switching display areas on a page-by-page basis via a network, includes: a drawing data receiving unit for receiving drawing data drawn in the display area from the plurality of devices; a writing unit for writing the plurality of received drawing data into one drawable area or the same page as the device; a drawing data moving unit for moving at least a part of the drawing data of the drawable area to an area corresponding to the display area of a predetermined device, and arranging the drawing data of the plurality of devices in the area, or arranging the drawing data of the plurality of pages in the area; and a drawing data transmitting unit for transmitting the drawing data of each device arranged in the area by the drawing data moving unit to the predetermined device.SELECTED DRAWING: Figure 5

Description

Technical Field

[0001] The present invention relates to an information processing apparatus and an information display method.

Background Art

[0002] In a remote conference between a plurality of bases, devices such as an electronic blackboard that allows a user to perform handwritten input are used. Also, in a remote conference or the like between different bases, a method is known in which while each base device shares drawing data such as handwritten data displayed at each base, handwritten input is performed on the same screen from each base.

[0003] In a remote conference, a technique has been devised for a user at each base to efficiently grasp the content of handwritten input (see, for example, Patent Document 1). Patent Document 1 discloses a configuration in which the whole is reduced and displayed so that drawing data within a region corresponding to the screen size of a terminal and drawing data outside the region can be viewed on one screen.

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, the conventional technology has a problem that the drawing data becomes very small in order to reduce and display the whole of the drawing data at each base. For example, when displaying a list of the whole drawing data drawn on a plurality of pages, there are gaps within the page, so there are many blank parts, and when the whole is reduced and displayed, the drawing data becomes small. When a user at each base handwrites at an arbitrary position in a vast drawable area, if the positions of the drawing data at each base are far apart, there are many blank parts, and when the whole is reduced and displayed, the drawing data becomes very small.

[0005] In view of the above problems, the present invention provides a technique capable of suppressing the reduction of drawing data at each base and displaying the drawing data at each base.

Means for Solving the Problems

[0006] In view of the above problems, the present invention is an information processing apparatus capable of communicating with a plurality of devices that display an arbitrary display area of a drawable area or switch the display area in page units via a network, the apparatus including: a drawing data receiving unit that receives drawing data drawn in the display area from the plurality of devices; a writing unit that writes the received plurality of drawing data into one of the drawable areas or the same page as the device; a drawing data moving unit that moves at least a part of the drawing data of the drawable area to an area corresponding to the display area of a predetermined device and arranges the drawing data of the plurality of devices in the area, or arranges the drawing data of a plurality of pages in the area; and a drawing data transmitting unit that transmits the drawing data of each device arranged in the area by the drawing data moving unit to the predetermined device.

Effect of the Invention

[0007] It is possible to suppress the reduction of the drawing data of each site and display the drawing data of each site.

Brief Description of the Drawings

[0008]

Figure 1

Figure 2

Figure 3

Figure 4A

Figure 4B

Figure 5

Figure 6

Figure 7A

Figure 7B

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15A

Figure 15B

Figure 16A

Figure 16B

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Figure 23

Figure 24

Figure 25

Figure 26

Figure 27

Figure 28

Figure 29

Figure 30

Figure 31

Figure 32

Figure 33

Figure 34

Figure 35

Figure 36

Figure 37

Figure 38

Figure 39

Figure 40

Figure 41

Figure 42

Figure 43

Figure 44

Figure 45

Figure 46

Figure 47

Figure 48

Figure 49

Figure 50

Figure 51

Figure 52

Figure 53

Figure 54

Figure 55

Figure 56

Figure 57

Figure 58

Figure 59

Figure 60

Figure 61

Mode for Carrying Out the Invention

[0009] Hereinafter, as an example of a mode for carrying out the present invention, a remote conferencing system and an information display method performed by the remote conferencing system will be described with reference to the drawings.

[0010] [First Embodiment] <Supplementary Explanation Regarding Confirmation of Display Contents Displayed at Multiple Bases> Devices such as electronic blackboards use one display area of a screen as one page. When there is no empty space left in the display area, the user switches pages and writes by hand on a new page. In this case, as a method of trying to display both the first page and the added page simultaneously, there is a display method of shrinking and arranging them so that each page fits within one screen. Also, such operations could sometimes be very cumbersome.

[0011] Also, there are known devices that do not have page breaks and allow handwriting in an infinite area (it does not actually have to be infinite, as long as it is larger than the display area) while scrolling the display area in any direction. With such devices, users at each base can scroll the display area of the device and write by hand in a preferred location.

[0012] In any of the above cases, it is difficult to display all the handwritten data written at each base on one screen. Therefore, when any user wants to view the handwritten data of each base, the user has to manually arrange and display multiple pages or scroll the display area to view it, which takes time for this operation.

[0013] Also, in the prior art, although the whole is reduced and displayed so that the entire handwritten data can be seen, if the positions of the handwritten data written by each user are far apart, there will be many blank parts, and when the whole is reduced and displayed, the handwritten data becomes very small and difficult to view.

[0014] Therefore, if the device cuts out the handwritten data of each base and moves it to the area corresponding to the display area of the device, there is a possibility that all the handwritten data can be displayed in the display area.

[0015] On the other hand, depending on the number of characters of each user or the type of device (electronic blackboard, tablet PC3 (Personal Computer), whiteboard in VR space, etc.), there may be cases where large characters are frequently written, and simply combining them may not fit on the display screen.

[0016] To address this, in the present embodiment, when the handwritten data input from each device protrudes from the display area (screen size) of the device and is displayed with respect to the drawable area on the memory shared by a plurality of devices, or when the handwritten data is displayed on a plurality of pages, all the handwritten data is arranged and displayed in the area corresponding to the display area of the device.

[0017] FIG. 1 is a diagram for explaining an outline of a process in which a conference server according to the present embodiment displays a list of drawing data of devices at a plurality of sites. FIG. 1(a) shows a display area 201 drawn by the device at site 1 in the drawable area 217. At site 1, two handwritten data 203 of "あ", a square 205, and a star mark 204 are drawn. The user can indicate in the display area 201 which part of the drawable area 217 is to be displayed. The drawable area 217 is wider than the display area of the device and can be said to be infinite within the range allowed by the memory. The drawable area 217 exists in the memory of the conference server, and each device can display any area within this area or add drawing data to the display area.

[0018] FIG. 1(b) shows a display area 202 displayed by the device at site 2 in the drawable area 217. At site 2, the user is scrolling the display area 201 of site 1 slightly upward and to the right and writing by hand. At site 2, three handwritten data 208 of "い" and a graphic of an arrow 206 are drawn.

[0019] When the user at site 1 or 2 wants to display a list of the drawing data of sites 1 and 2, the user presses (clicks, taps, touches with the pen tip) the overall display button 210. In this case, the conference server groups the drawing data of each site. FIG. 1(c) shows the grouped drawing data. The grouping is performed, for example, by specifying all the drawing data of one site with a circumscribing rectangle.

[0020] Next, the conference server moves Group 1 and Group 2 closer. Either one can be moved, but since Group 1 is on the far left, its position is fixed, and the conference server moves Group 2 closer to Group 1. For example, the upper left corner of Group 2 is moved to the upper right corner of Group 1.

[0021] Figure 1(d) shows the drawing data of Groups 1 and 2 after the movement. Figure 1(d) is, for example, the display area 201 of the device at Base 1. In this way, by moving Group 2 close to Group 1, the electronic blackboard can list and display all the drawing data in the display area 201 of the device.

[0022] <Regarding Terms> Drawing data is data drawn in the display area of a device. Drawing includes not only handwriting but also displaying images acquired from the outside. Drawing data can be, for example, handwriting data, text data (fonts), graphics, screen data, images, etc., as long as it is displayable data.

[0023] A device can be any device that connects to the conference server to participate in a remote conference and display drawing data. In this embodiment, the device can be, for example, an electronic blackboard, a tablet PC, a notebook PC, a smartphone, a projector, etc., but is not limited to these.

[0024] An electronic blackboard is a device that uses a touch panel to display handwritten characters, graphics, etc. in real time on a display or the like. The electronic blackboard may have a function to connect to a network or a function to convert handwritten characters into font data using OCR.

[0025] The display area is the maximum range where a device can display drawing data and is specified by the number of pixels in the vertical and horizontal directions. Also, the drawable area is the area on the memory where drawing data is stored and is wider than the display area. The drawable area is held by the device and the conference server respectively.

[0026] <Example of System Configuration> Referring to FIG. 2, the system configuration of the remote conferencing system 300 will be described. FIG. 2 is a diagram showing an example of the system configuration of the remote conferencing system 300. In the remote conferencing system 300, devices at each site are communicably connected via the conference server 1 and the communication network N. The devices are, for example, an electronic blackboard and a tablet PC 3, but this is just an example.

[0027] The communication network N is, for example, the Internet, but may also be an on-premises network such as a corporate network. The number of devices and the number of sites are just examples, and any number of devices may be used as long as there are two or more. Also, a plurality of devices may be arranged at one site.

[0028] The conference server 1 is an information processing system having one or more information processing devices. The conference server 1 manages conference information and the like, and manages the start and end of the conference, the login of participants, etc. The service provided by the conference application executed on the conference server 1 is called a conference service.

[0029] The electronic blackboard 2 and the tablet PC 3 are examples of devices operated by conference participants (users). The devices are not limited to the electronic blackboard 2 and the tablet PC 3, and may be any information processing terminal having a communication function, such as a PC, a smartphone, or a PDA (Personal Digital Assistant), on which the conference application operates.

[0030] The device preferably has a touch panel. The user can write by hand with an electronic pen or a fingertip. Even if the device does not have a touch panel, the user can write by hand with a pointing device such as a mouse. Also, in addition to handwritten data, the device can display drawing data such as text converted from handwritten data, text input by voice, graphics, and the screens of PCs, etc., and these drawing data can also be shared between sites.

[0031] In the configuration as shown in FIG. 2, the electronic blackboard 2 and the tablet PC 3 are connected to the same URL for meetings. The URL and password are distributed to the participants in advance by e-mail or the like. The drawing data displayed on the electronic blackboard 2, the surrounding video data input from the camera, and the audio data input from the microphone are transmitted to the conference server 1, and the conference server 1 transmits them to the tablet PC 3. The tablet PC 3 displays the received drawing data and video data and outputs the audio data. Similarly, the drawing data displayed on the tablet PC 3, the video data input from the camera, and the audio data input from the microphone are transmitted to the conference server 1, and the conference server 1 transmits them to the electronic blackboard 2. The electronic blackboard 2 displays the received drawing data and video data and outputs the audio data. An online meeting is held by repeating this process between the electronic blackboard 2 and the tablet PC 3.

[0032] <Hardware Configuration Example> <<Electronic Blackboard>> FIG. 3 is a hardware configuration diagram of an example of the electronic blackboard 2. As shown in FIG. 3, the electronic blackboard 2 includes a CPU (Central Processing Unit) 401, a ROM (Read Only Memory) 402, a RAM (Random Access Memory) 403, an SSD (Solid State Drive) 404, a network I / F 405, and an external device connection I / F (Interface) 406.

[0033] Among these, the CPU 401 controls the operation of the entire electronic blackboard 2. The ROM 402 stores programs used for starting the OS such as the IPL (Initial Program Loader). The RAM 403 is used as a work area for the CPU 401. The SSD 404 stores various data such as programs for the electronic blackboard 2. The network I / F 405 controls communication with the communication network N. The external device connection I / F 406 is an interface for connecting various external devices. The external devices in this case are, for example, a USB (Universal Serial Bus) memory 430 and external devices (a microphone 440, a speaker 450, a camera 460).

[0034] In addition, the electronic blackboard 2 includes a capture device 411, a GPU 412, a display controller 413, a touch sensor 414, a sensor controller 415, an electronic pen controller 416, a short-range communication circuit 419, an antenna 419a of the short-range communication circuit 419, a power switch 422, and selection switches 423.

[0035] Among these, the capture device 411 causes the display information of the external PC 470 to be displayed as a still image or a moving image. The GPU (Graphics Processing Unit) 412 is a semiconductor chip specialized in handling graphics. The display controller 413 controls and manages the screen display in order to output the output image from the GPU 412 to the display 480 or the like. The touch sensor 414 detects that an electronic pen 490, the user's hand H, etc. have touched the display 480. The sensor controller 415 controls the processing of the touch sensor 414. The touch sensor 414 detects the input coordinates by an infrared blocking method. This detection of the input coordinates is performed using two light emitting and receiving devices installed at both upper ends of the display 480. The light emitting element (laser) in the light emitting and receiving device rotates and scans an infrared beam in a range of 90 degrees parallel to the display 480. The infrared beam is reflected by a reflecting member provided around the display 480. The light receiving element receives the light that returns on the same optical path as the optical path of the emitted light. The light emitting and receiving device, which is the touch sensor 414, outputs the position where the infrared ray is blocked by the object (the position on the light receiving element) to the sensor controller 415, and the sensor controller 415 specifies the coordinate position, which is the contact position of the object, from the infrared blocking position information on these two light receiving elements. The electronic pen controller 416 determines whether the touch on the display 480 is at the pen tip or the pen butt by communicating with the electronic pen 490 via Bluetooth (registered trademark. Hereinafter, omitted). The electronic pen 490 is provided with a switch for switching between a mode of operating with the pen tip and a mode of operating with the pen butt, and is configured to transmit the set mode information via Bluetooth. When in the pen tip mode, the electronic blackboard 2 draws a line at the position of the coordinate sequence traced by the touch, and when in the pen butt mode, it functions as an eraser and erases the line drawn at the position of the coordinate sequence traced by the touch. The short-range communication circuit 419 is a communication circuit such as NFC (Near Field Communication) or Bluetooth. The power switch 422 is a switch for switching the ON / OFF of the power supply of the electronic blackboard 2.The selection switches 423 are, for example, a group of switches for adjusting the brightness, color tone, etc. of the display of the display 480.

[0036] Furthermore, the electronic blackboard 2 includes a bus line 410. The bus line 410 is an address bus, a data bus, etc. for electrically connecting each component such as the CPU 401 shown in FIG. 3.

[0037] Note that the contact sensor 414 is not limited to the infrared blocking method, and may be a capacitive touch panel that identifies the contact position by detecting a change in capacitance. The contact sensor 414 may also be a resistive film type touch panel that identifies the contact position by the voltage change of two opposing resistive films. The contact sensor 414 may also be an electromagnetic induction type touch panel that detects the electromagnetic induction generated when the contact object touches the display unit and identifies the contact position. In addition, the contact sensor 414 may use various detection means. Also, the electronic pen controller 416 may be configured to determine the presence or absence of touch not only at the tip and the butt of the electronic pen 490, but also at the part held by the user of the electronic pen 490 and other parts of the electronic pen.

[0038] <<Conference Server>> FIG. 4A is a diagram showing an example of the hardware configuration of the conference server 1. As shown in FIG. 4A, the conference server 1 is constructed by a computer 500 and includes a CPU 501, a ROM 502, a RAM 503, an HD (Hard Disk) 504, an HDD (Hard Disk Drive) controller 505, a display 506, an external device connection I / F (Interface) 508, a network I / F 509, a bus line 510, a keyboard 511, an optical drive 514, and a media I / F 516.

[0039] Among these, the CPU 501 controls the operation of the entire conference server 1. The ROM 502 stores programs used for driving the CPU 501 such as the IPL. The RAM 503 is used as a work area for the CPU 501. The HD 504 stores various data such as programs. The HDD controller 505 controls the reading or writing of various data to and from the HD 504 according to the control of the CPU 501. The display 506 displays various information such as a cursor, menu, window, characters, or images. The external device connection I / F 508 is an interface for connecting various external devices. External devices in this case are, for example, a USB (Universal Serial Bus) memory, a printer, and the like. The network I / F 509 is an interface for performing data communication using a network. The bus line 510 is an address bus, a data bus, etc. for electrically connecting each component such as the CPU 501 shown in FIG. 4A.

[0040] Also, the keyboard 511 is a type of input means having a plurality of keys used for inputting characters, numerical values, or various instructions. The optical drive 514 controls the reading or writing of various data to and from the optical storage medium 513 as an example of a removable recording medium. Note that the optical storage medium 513 may be a CD, DVD, Blu-ray (registered trademark), or the like. The media I / F 516 controls the reading or writing (storage) of data to and from the recording medium 515 such as a flash memory. <<Tablet PC>> FIG. 4B is a diagram showing the hardware configuration of an example of the tablet PC 3. As shown in FIG. 4B, the tablet PC 3 is constructed by a computer 520 and includes a CPU 521, a ROM 522, a RAM 523, an HD (Hard Disk) 524, an HDD (Hard Disk Drive) controller 525, a display 526, an external device connection I / F (Interface) 527, a bus line 528, a network I / F 529, a pointing device 530, and a media I / F 531. Note that the pointing device 530 is a touch panel. Among these, the CPU 521 controls the operation of the entire tablet PC 3. The ROM 522 stores programs used for driving the CPU 521 such as the IPL. The RAM 523 is used as a work area for the CPU 521. The HD 524 stores various data such as programs. The HDD controller 525 controls the reading or writing of various data to and from the HD 524 according to the control of the CPU 521. The display 526 displays various information such as a cursor, menu, window, characters, or images. The external device connection I / F 527 is an interface for connecting various external devices. The external devices in this case are, for example, a camera 540, a microphone 541, a speaker 542, a USB (Universal Serial Bus) memory, a printer, and the like. The network I / F 529 is an interface for performing data communication using a network. The bus line 528 is an address bus, a data bus, etc. for electrically connecting each component such as the CPU 521 shown in FIG. 4B. The pointing device 530 is a type of input means for selecting and executing various instructions, selecting a processing target, moving a cursor, and the like. The media I / F 531 controls the reading or writing (storage) of data to and from the recording medium 532 such as a flash memory.

[0041] <Regarding functions> Next, with reference to FIG. 5, the functional configuration of the remote conference system 300 will be described. FIG. 5 is an example of a functional block diagram for separately explaining the functions of the conference server 1, the electronic blackboard 2, and the tablet PC 3 in the remote conference system 300 in blocks.

[0042] <<Conference server>> The conference server 1 has a drawing data receiving unit 53, a drawing data grouping processing unit 54, a conference control unit 55, a drawing data moving unit 56, a drawing data transmitting unit 57, a device authentication unit 58, a user authentication unit 59, and a writing unit 66. Each of these units of the conference server 1 is a function or a means for realizing the operation by an instruction from the CPU 501 according to a program expanded from the HD 504 to the RAM 503 by any of the components shown in FIG. 4A.

[0043] The drawing data receiving unit 53 is a means for receiving drawing data drawn in the display area from a plurality of devices. Specifically, the drawing data receiving unit 53 receives drawing data such as handwritten data and graphics transmitted from the electronic blackboard 2 or the tablet PC 3.

[0044] The drawing data grouping processing unit 54 is a means for grouping the drawing data written in the drawable area for each device. Specifically, the drawing data grouping processing unit 54 specifies the drawing data with a circumscribed rectangle or the like, and groups the drawing data for each of the electronic blackboard 2 and the tablet PC 3.

[0045] The drawing data moving unit 56 is a means for moving at least a part of the drawing data in the drawable area and arranging the drawing data of a plurality of bases in an area corresponding to the display area of a predetermined device. Alternatively, the drawing data moving unit 56 is a means for arranging the drawing data of a plurality of pages in an area corresponding to the display area of a predetermined device. Specifically, the drawing data moving unit 56 moves the drawing data so that the grouped drawing data enters an area corresponding to the display area of the device.

[0046] The drawing data transmitting unit 57 is a means for transmitting the drawing data of each base arranged by the drawing data moving unit 56 in an area corresponding to the display area of a predetermined device to the predetermined device. Specifically, the drawing data transmitting unit 57 transmits the drawing data transmitted from each base to the devices of other bases. When the overall display button is pressed, the drawing data transmitting unit 57 transmits the entire drawing data including the drawing data of the moved base to the device that pressed the overall display button.

[0047] The conference control unit 55 generates and registers a conference (conference ID) according to a request from a device participating in the conference. The conference control unit 55 distributes voice data and image data (including video) received from a device participating in the conference to other devices (distribution of data other than drawing data).

[0048] The device authentication unit 58 authenticates the electronic blackboard 2 based on the device information transmitted from the electronic blackboard 2. The user authentication unit 59 authenticates a user based on the user information (for example, user ID and password) transmitted from the tablet PC 3.

[0049] The writing unit 66 is a means for writing a plurality of pieces of drawing data received from the electronic blackboard 2 or the tablet PC 3 into one drawable area or the same page as the electronic blackboard 2. Specifically, the writing unit 66 writes the drawing data transmitted from a plurality of devices into the drawable area 217 managed by the conference server 1. The drawable area 217 is common to a plurality of devices.

[0050] <<Function of the electronic blackboard>> The electronic blackboard 2 includes a drawing data transmission unit 41, a drawing data reception unit 42, a data storage unit 43, a display control unit 44, a contact position detection unit 45, a device information transmission unit 46, a drawing data generation unit 95, and a LAN communication control unit 96. Each of these units included in the electronic blackboard 2 is a function or a means for operating realized by any of the components shown in FIG. 3 according to an instruction from the CPU 401 according to a program expanded from the SSD 404 to the RAM 403. Note that the data storage unit 43 is constructed in the SSD 404, the RAM 403, etc.

[0051] The drawing data transmission unit 41 transmits drawing data such as handwritten data and graphics to the conference server 1.

[0052] The drawing data reception unit 42 receives drawing data such as handwritten data and graphics transmitted from the conference server 1.

[0053] The display control unit 44 performs display of drawing data such as handwritten data and graphics, display of meeting materials, display of drawing data received from the meeting server 1, and update of the display area accompanying a scroll operation, etc.

[0054] The contact position detection unit 45 detects the contact position of the user's finger or pen. The device information transmission unit 46 transmits device information to the meeting server 1. The data storage unit 43 stores the setting data and device information of the electronic blackboard 2. The drawing data generation unit 95 generates drawing data from the coordinate point sequence input from the contact position detection unit 45 when the user writes by hand, or generates graphics such as circles and rectangles. The LAN communication control unit 96 is a means for transmitting and receiving data other than drawing data to and from the meeting server 4. Specifically, the LAN communication control unit 96 connects to a communication network N such as Ethernet (registered trademark) and performs transmission and reception of data with other devices via the communication network N.

[0055] <<Tablet PC>> The tablet PC 3 has a drawing data transmission unit 47, a drawing data reception unit 48, a data storage unit 49, a display control unit 50, a contact position detection unit 51, and a user information transmission unit 52. Each of these units of the tablet PC 3 is a function or means realized by any of the components shown in Fig. 4B operating according to instructions from the CPU 521 developed from the HD 524 to the RAM 523. Note that the data storage unit 49 is constructed in the HD 524, the RAM 523, etc.

[0056] The drawing data transmission unit 47 transmits drawing data such as handwritten data and graphics to the meeting server 1.

[0057] The drawing data reception unit 48 receives drawing data such as handwritten data and graphics transmitted from the meeting server 1.

[0058] The display control unit 50 performs display of drawing data such as handwritten data and graphics, display of meeting materials, display of drawing data received from the meeting server 1, update of the screen accompanying a scroll operation, etc.

[0059] The contact position detection unit 51 detects the contact position of the user's finger or pen. The user information transmission unit 52 transmits user information to the conference server 1. The data storage unit 49 stores the setting data of the tablet PC 3 and the data of the applications used by the user.

[0060] <Start of the conference> The electronic blackboard 2 and the tablet PC 3 are connected to the conference server 1 to start a conference. (1) When the power of the electronic blackboard 2 at Site 1 is turned on automatically or by the user's operation, the electronic blackboard 2 is connected to the conference server 1. If the electronic blackboard 2 is registered in advance in the conference schedule or the like, it participates in that conference. The electronic blackboard 2 transmits device information to the conference server 1, and the conference server 1 authenticates the electronic blackboard 2 with the device information and permits participation in the conference upon successful authentication. (2) By the operation of the user at Site 2, the tablet PC 3 is connected to the conference server 1. The tablet PC 3 designates a conference with a URL or the like and connects to this conference. At this time, the tablet PC 3 transmits user information to the conference server 1, and the conference server 1 permits the user to participate in the conference upon successful authentication. (3) Subsequently, when the user at Site 1 starts the whiteboard application of the electronic blackboard 2, the electronic blackboard 2 displays the screen of the whiteboard application on the display. The display area of the screen of the whiteboard application is the display area 201. The electronic blackboard 2 transmits a whiteboard start command including the number of display pixels (the number of pixels in the vertical direction and the number of pixels in the horizontal direction) of the display area 201 to the conference server 1 to enter the drawing data sharing mode with devices at other sites. When the conference server 1 receives the whiteboard start command, it stores in the memory the number of display pixels transmitted together with the whiteboard start command in association with the device information of the electronic blackboard 2. The conference server 1 generates a drawable area 217 with an arbitrary size larger than the size of the number of display pixels. Generating the drawable area 217 means securing a vast area for writing drawing data on the memory. (4) When the user at Site 2 also launches the whiteboard application on the tablet PC 3, the tablet PC 3 displays the screen of the whiteboard application on the display. The display area 202 of the tablet PC 3 is set to the maximum screen size that the whiteboard application screen can take. The tablet PC 3 sends a whiteboard start command including the number of display pixels in the display area 202 to the conference server 1, and enters the drawing data sharing mode with devices at other sites. When the conference server 1 receives the whiteboard start command, it stores in memory the number of display pixels sent together with the whiteboard start command, associating it with the device information of the tablet PC 3. (5) The display areas of the electronic whiteboard 2 and the whiteboard application displayed on the tablet PC 3 are managed in association with positions on the memory of the conference server 1, and the display area moves by scrolling the display area on the electronic whiteboard 2 or the tablet PC 3. The user can change the display areas 201 and 202 of the electronic whiteboard 2 and the tablet PC 3. (6) When the user writes by hand on the electronic whiteboard 2 at Site 1, the contact position detection unit 45 detects the contact position, and the display control unit 44 generates and displays handwritten data at that position. Also, when the user selects a figure from the figure list displayed by pressing a figure icon in the menu bar described later, this figure is generated and displayed.

[0061] FIG. 6 shows an example of the figure list 212 displayed by pressing the figure icon 211. The menu bar 215 has the figure icon 211 and the full display button 210. The user can select any figure from the figure list 212. The electronic whiteboard 2 displays the selected figure. The drawing data transmission unit 47 of the electronic whiteboard 2 or the tablet PC 3 sequentially sends drawing data such as handwritten data and figures to the conference server 1. The conference server 1 writes the received drawing data into the drawable area 217 of the whiteboard in memory. When the characters "あ", a square, and a star mark are drawn, the drawing data of the drawable area 217 in the memory of the conference server 1 is shown in FIG. 7A.

[0062] FIG. 7A shows the drawing data of the drawable area 217 in the memory of the conference server 1. The drawing data in FIG. 7A is two handwritten data 203 of "a", a square 205, and a star mark 204. In addition, the display area 201 of the electronic blackboard 2 is shown in FIG. 7A. The display area 201 may vary for each device, and the conference server 1 holds the current display area of each device.

[0063] In addition, the conference server 1 transmits the received drawing data to the tablet PC 3 at site 2 that is participating in the same conference. The tablet PC 3 at site 2 receives the drawing data and displays the drawing data.

[0064] The user at site 2 wants to handwrite information related to the drawing data input at site 1 in the empty space on the right side of the star mark 204. However, since the empty area is narrow, the user at site 2 scrolls the drawable area 217 to the left to expand the empty area and handwrites there. Note that the scroll direction is performed with respect to the drawable area 217 of the whiteboard rather than the display area, so it is the direction in which the drawable area 217 is moved while fixing the display area.

[0065] The contact position detection unit 51 of the tablet PC 3 detects the leftward scroll operation by the user with the pen, and the display control unit 50 moves the display area. In the tablet PC 3 at site 2, when the user handwrites or displays a figure with the pen, the contact position detection unit 51 detects the contact position of the pen, and the display control unit 50 generates and displays the handwritten data and the figure. The drawing data transmission unit 47 sequentially transmits this drawing data to the conference server 1. The conference server 1 writes the drawing data received from the tablet PC 3 into the drawable area 217 of the whiteboard on the memory.

[0066] When the user handwrites the handwritten data 208 of "i" and displays the arrow 206 on the tablet PC 3, FIG. 7B shows the drawing data of the drawable area 217 on the memory of the conference server 1 after the tablet PC 3 transmits these drawing data.

[0067] In addition, FIG. 7B shows the display area 202 of the tablet PC 3. When the conference server 1 receives the drawing data from site 2, the conference server 1 writes the drawing data to the drawable area 217 of the whiteboard in the memory. In addition, the conference server 1 transmits the received drawing data to the electronic blackboard 2 at site 1.

[0068] FIG. 8 shows the display area 201 of the electronic blackboard 2 at site 1. The electronic blackboard 2 at site 1 receives the drawing data. However, since the handwritten data 208 and the arrow 206 are drawn on the tablet PC 3 outside the display area of the electronic blackboard 2, the handwritten data 208 and the arrow 206 displayed on the tablet PC 3 are not displayed in the display area 201 of the electronic blackboard 2 at site 1.

[0069] Therefore, the user at site 1 presses the overall display button 210 in order to view all the drawing data on the whiteboard on the electronic blackboard 2. A menu bar 215 is displayed at a fixed position in the display area 201 on the electronic blackboard 2, and the menu bar 215 has the overall display button 210. When the user at site 1 presses the overall display button 210, the contact position detection unit 45 detects the pressing of the overall display button 210. The electronic blackboard 2 transmits an overall display command (a request for a list display of drawing data) to the conference server 1.

[0070] When the conference server 1 receives the overall display command, the drawing data grouping unit 54 of the conference server 1 groups the drawing data drawn on the electronic blackboard 2 and the drawing data drawn on the tablet PC 3, respectively.

[0071] FIG. 9 shows Group 1 of the drawing data on the electronic blackboard 2 and Group 2 of the drawing data on the tablet PC 3. Note that the grouping is, for example, to determine the circumscribed rectangles 221 and 222 of all the drawing data in each device. Groups 1 and 2 may overlap. The circumscribed rectangles 221 and 222 in FIG. 9 are for explanation purposes, and the circumscribed rectangles 221 and 222 are not drawn in the memory of the conference server 1.

[0072] Since the overall display button 210 has been pressed on the electronic blackboard 2, the drawing data moving unit 56 fixes group 1 of the electronic blackboard 2 and moves the other group 2 close to group 1. For example, when the overall display button 210 is pressed on the tablet PC 3, the drawing data moving unit 56 may fix group 2 of the tablet PC 3 and move the other group 1 close to group 2. However, regardless of which device the overall display button 210 is pressed on, it is always possible to fix the leftmost group 1 and move the other group 2 close to group 1, or fix the rightmost group 2 and move the other group 1 close to group 2. Alternatively, the center of gravity of all the drawing data in the drawable area 217 may be obtained, and the group of the device on which the drawing data closest to the center of gravity is drawn may be fixed and the other groups may be moved.

[0073] FIG. 10 is a diagram for explaining the movement of groups. In FIG. 10, a rectangular area 223 circumscribing each group is shown so as to include each group 1, 2. The width direction of the display area 201 is the X-axis, and the height direction is the Y-axis (the same applies to the following figures). The coordinates of the upper left corner of this rectangular area 223 are taken as the origin (0, 0). Also, assuming that the coordinates of groups 1 and 2 are as follows with respect to (0, 0). The coordinates of the upper left corner of group 1 are (0, y1) The coordinates of the upper right corner of group 1 are (x1, y1) The coordinates of the upper left corner of group 2 are (x2, 0) The coordinates of the lower left corner of group 2 are (x2, y2) The drawing data moving unit 56 of the conference server 1 calculates the horizontal distance (x2 - x1) from the coordinates (x2, 0) of the upper left corner of group 2 to the coordinates (x1, y1) of the upper right corner of group 1. Note that the distance is taken as the absolute value. To avoid the right end of group 1 and the left end of group 2 from overlapping and being difficult to see, the drawing data moving unit 56 subtracts a margin of α from the distance. Also, the drawing data moving unit 56 calculates the vertical distance (y1) from the coordinates (x2, 0) of the upper left corner of group 2 to the coordinates (x1, y1) of the upper right corner of group 1.

[0074] In the drawable area 217, the drawing data moving unit 56 moves the drawing data of group 2 leftward by a distance of (x2 - x1 - α) and downward by a distance of (y1). Alternatively, in the drawable area 217, the drawing data moving unit 56 moves the drawing data of group 1 rightward by a distance of (x2 - x1 - α) and upward by a distance of (y1). Note that the drawing data moving unit 56 holds the drawing data of each group before movement or at least its coordinates. This is because the user at site 2 has been moved without consent and the user at site 1 also needs to return to the original state.

[0075] The overall circumscribed rectangles of groups 1 and 2 after the movement are an example of the "area corresponding to the display area 201". The drawing data transmission unit 57 of the conference server 1 transmits the moved drawing data to the electronic blackboard 2, and the drawing data reception unit 42 of the electronic blackboard 2 at site 1 receives the drawing data. The display control unit 44 of the electronic blackboard 2 displays the drawing data.

[0076] FIG. 11 shows the drawing data after the movement displayed on the electronic blackboard 2. Inside the display area 201 of the electronic blackboard 2, the drawing data (handwritten data 208 and arrow 206) drawn on the tablet PC 3 is displayed. In this way, by the conference server 1 moving group 2 close to group 1, all the drawing data of each site can be displayed in a list in the display area 201 of the electronic blackboard 2 without reducing the drawing data. The overall display button has changed to an overall display cancel button, and when the user presses the overall display cancel button, it returns to the original display area 201. In this case, the electronic blackboard 2 may receive the drawing data of the display area 201 from the conference server 1, or may acquire the drawing data of the display area 201 stored in the data storage unit 43.

[0077] <Operation or process> FIG. 12 is a sequence diagram for explaining the process in which the conference server 1 moves the drawing data of the tablet PC 3 and displays it in the display area 201 of the electronic blackboard 2. Note that the electronic blackboard 2 and the tablet PC 3 are connected to the same conference (session). The conference server 1 authenticates both using the device information of the electronic blackboard 2 and the user ID of the user operating the tablet PC 3, etc., and transmits the drawing data to the destination at the IP address of the electronic blackboard 2 associated with the device information of the electronic blackboard 2 and the IP address of the tablet PC 3 associated with the user ID.

[0078] S1: The user of the electronic blackboard 2 writes by hand on the electronic blackboard 2 with a pen.

[0079] S2: The contact position detection unit 45 of the electronic blackboard 2 detects the contact position of the pen. If the contact position is the graphic icon 211 of the menu bar 215, the graphic may be selected. The display control unit 50 displays drawing data such as handwritten data and graphics in the display area 201 of the electronic blackboard 2.

[0080] S3: The drawing data transmission unit 41 of the electronic blackboard 2 sequentially transmits the drawing data to the conference server 1. Sequentially means, for example, for each drawing line, such as from pen down to pen up.

[0081] S4: The drawing data reception unit 53 of the conference server 1 receives the drawing data, and the writing unit 66 writes the drawing data to the whiteboard (drawable area 217) in the memory. Coordinates in the drawable area 217 are associated with the drawing data. Also, it is recorded from which device each drawing data was transmitted.

[0082] S5: Next, the drawing data transmission unit 57 of the conference server 1 transmits the drawing data received from the electronic blackboard 2 to the tablet PC 3, preferably immediately afterwards.

[0083] S6: The drawing data receiving unit 48 of the tablet PC 3 receives the drawing data. The display control unit 50 draws the drawing data included in the current display area 202 of the tablet PC 3, but does not draw the drawing data not included in the display area 202. The drawing data is retained in the data storage unit 49 even if it is not drawn.

[0084] S7 - S12: The user of the electronic blackboard 2, the electronic blackboard 2, the conference server 1, and the tablet PC 3 repeatedly execute steps S1 - S6 each time the user causes drawing data to be drawn.

[0085] S13: Next, the user of the tablet PC 3 wants to handwrite drawing data near the drawing data drawn on the electronic blackboard 2, but there is no empty space. Therefore, the user of the tablet PC 3 scrolls the display area 202.

[0086] S14: The contact position detection unit 51 of the tablet PC 3 detects the scroll by the pen, the direction, and the scroll amount. The display control unit 50 moves the display area 202 in the tablet PC 3 according to the detected scroll direction and scroll amount.

[0087] S15: The user of the tablet PC 3 handwrites on the tablet PC 3 with a pen.

[0088] S16: The contact position detection unit 51 of the tablet PC 3 detects the contact position of the pen. If the contact position is the graphic icon 211 of the menu bar 215, the graphic may be selected. The display control unit 50 displays drawing data such as handwritten data and graphics in the display area 202 of the tablet PC 3.

[0089] S17: The drawing data transmission unit 47 of the tablet PC 3 sequentially transmits the drawing data to the conference server 1.

[0090] S18: The drawing data receiving unit 53 of the conference server 1 receives the drawing data, and the writing unit 66 writes the drawing data to the whiteboard (drawable area 217) in the memory. Coordinates in the drawable area 217 are associated with the drawing data. Also, it is recorded from which device each drawing data is transmitted.

[0091] S19: Next, the drawing data transmission unit 57 of the conference server 1 preferably transmits the drawing data received from the tablet PC 3 to the electronic blackboard 2 immediately.

[0092] S20: The drawing data receiving unit 42 of the electronic blackboard 2 receives the drawing data. The display control unit 44 draws the drawing data included in the current display area 201 of the electronic blackboard 2, but does not draw the drawing data not included in the display area 201. The drawing data is held in the data storage unit 43 even if it is not drawn.

[0093] S21 - S26: The user of the tablet PC 3, the tablet PC 3, the conference server 1, and the electronic blackboard 2 repeatedly execute steps S15 - S20 each time the user causes drawing data to be drawn. The user may scroll the display area 202.

[0094] S27: Next, since the user of the electronic blackboard 2 also wants to display the drawing data drawn on the tablet PC 3 on the electronic blackboard 2, the user presses the overall display button 210 with a pen. When there are three or more bases, the user may be able to press the overall display button 210 after designating an arbitrary base. If not designated, the drawing data of all bases is listed.

[0095] S28: The contact position detection unit 45 of the electronic blackboard 2 detects that the position where the pen has contacted is the overall display button 210, and the drawing data transmission unit 41 transmits an overall display command to the conference server 1.

[0096] S29: The drawing data receiving unit 53 of the conference server 1 receives the overall display command. Since the overall display command is received, the drawing data grouping processing unit 54 groups the drawing data of each device.

[0097] S30: The drawing data moving unit 56 fixes, for example, the group of devices that transmitted the overall display command or the left group. The drawing data moving unit 56 moves the groups that are not the fixed group so that there is no empty space with the fixed group. As described above, the drawing data moving unit 56 calculates the amount of movement so that the upper right corner of the left group substantially coincides with the upper left corner of the right group, and moves the right group in the horizontal and vertical directions. By doing so, the gap between the drawing data of the groups is reduced, and the drawing data of the tablet PC 3 can also be displayed in the display area 201 of the electronic blackboard 2.

[0098] S31: The drawing data transmission unit 57 of the conference server 1 transmits the drawing data of the moved tablet PC 3 and the drawing data of the electronic blackboard 2 to the electronic blackboard 2. Since the drawing data drawn on the electronic blackboard 2 does not move, the drawing data drawn on the electronic blackboard 2 may not be transmitted.

[0099] S32: The drawing data receiving unit 42 of the electronic blackboard 2 receives at least the drawing data of the moved tablet PC 3 and the display control unit 44 displays it. Although there is no guarantee that all the drawing data will be displayed in the display area 201 of the electronic blackboard 2, it can be expected that the amount of drawing data that can be displayed in the display area 201 will increase compared to before pressing the overall display button 210.

[0100] Note that in this embodiment, an example in which the user of the electronic blackboard 2 presses the overall display button 210 has been described. However, when the user of the tablet PC 3 presses the overall display button 210, all the drawing data can similarly be displayed in the display area 202 of the tablet PC 3.

[0101] <Main effects> The conference server 1 of this embodiment can reduce the gap between the drawing data by moving one group close to the other group, and increase the possibility of displaying all the drawing data in a list in the display area of the device.

[0102] [Second Embodiment] In this embodiment, a conference server 1 that reduces or enlarges drawing data according to the character size of a device will be described.

[0103] <Regarding functions> FIG. 13 shows a functional block diagram of the conference server 1 in this embodiment. Note that since the functional block diagrams of the electronic blackboard 2 and the tablet PC 3 are the same as those in FIG. 5, only the functional block diagram of the conference server 1 is shown. The conference server 1 in FIG. 13 newly has a handwritten character detection unit 60 and a drawing data enlargement / reduction unit 61.

[0104] The handwritten character detection unit 60 is a means for detecting characters from grouped drawing data. Specifically, the handwritten character detection unit 60 detects handwritten characters from the drawing data. The handwritten characters may be numbers, and the handwritten character detection unit 60 analyzes the drawing data composed of a coordinate point sequence to detect handwritten characters.

[0105] The drawing data enlargement / reduction unit 61 is a means for obtaining the ratio of the average size of characters detected from the first group to the average size of characters detected from the second group, and when the ratio exceeds a threshold value, adjusting the sizes of the drawing data of the first group and the second group so that the ratio becomes the threshold value. Specifically, the drawing data enlargement / reduction unit 61 enlarges or reduces the grouped handwritten data, drawings, etc. of graphics so that the difference in character size between groups becomes small.

[0106] <Enlargement or reduction process> FIG. 14 shows the display area 201 of the electronic blackboard 2. As described in the first embodiment, group 2 has been moved near group 1. After such movement, the handwritten character detection unit 60 detects the handwritten characters of group 1 and the handwritten characters of group 2.

[0107] FIG. 15A is a diagram for explaining a method of detecting handwritten characters. In FIG. 15A, a dotted frame represents an undetermined segment, a solid line frame represents a new segment, and a thick arrow represents an index indicating the immediately previous undetermined segment. A segment consists of one or more strokes (drawing lines from pen-down to pen-up) and is a component of a character, and one character is composed of one or more segments.

[0108] In FIG. 15A, a segment that is more than two character sizes to the left of the current input position is defined as a determined segment. First, (a) in FIG. 15A shows the state of segment division when writing up to the third stroke of "な" (na) without a frame. Since each segment has a gap in the x-coordinate direction, it is divided into four undetermined segments.

[0109] When the next stroke (the 4th stroke of "na") is input to the initial state of Fig. 15A(a), as shown in (b), the 4th stroke of "na" becomes a new segment. As shown in (c), an index is set at the end of the undetermined segment (here, the undetermined segment of the 3rd stroke of "na"). Subsequently, the x - coordinate data of the undetermined segment and the new segment at the index position are compared. If they are at a position that meets a predetermined judgment criterion, as shown in (d), the two segments are combined to create a new new segment. Next, as shown in (e), the index is reset to the undetermined segment one before that. Similarly, as shown in (f) and (g), segment combination and index resetting are performed. In the state of (f), there is a clear gap between the new segment ("na") and the undetermined segment ("zhu"), so as shown in (g), they are not combined. Therefore, the index moves further forward. Since there is also a clear gap between the segment pointed to by the index ("zhu") and the previous segment ("mu"), they are not combined, and as shown in (h), the index is reset to the previous undetermined segment ("mu"). Since there is no undetermined segment before the segment "mu", as shown in (i), only the index moves forward, and finally when it reaches a state where it points to nothing (the index position is empty) before the first segment, the new segment is registered as a new undetermined segment. Further, if there are segments before, the judgment of this undetermined segment is made up to before the determined segment (in this example, within two characters from the latest segment).

[0110] FIG. 15B is a diagram for explaining a method of determining whether or not a segment has been confirmed. In FIG. 15B(a), the characters up to “桧な” have been written, and the characters in dotted lines thereafter are not written. Here, all segments are shown in dotted lines to indicate that they are all unconfirmed segments. (b) shows the state in which the next “し” has been written. Here, the segment “木” located at a position more than the threshold d away from the left end of the “し” segment is in a confirmed state. Similarly, when the “言” radical of “言” is input, the “卆” segment is confirmed because it is more than the threshold d away, the “な” segment is not confirmed because it is not more than the threshold d away, and when the “仁” of “言” is input, the “な” segment is confirmed because it is more than the threshold d away. In this way, as writing progresses, the segment located more than the threshold d away from the latest stroke is confirmed. The information of the segment that has become a newly confirmed segment from the unconfirmed segment is copied from the information of the unconfirmed segment and added to the list of confirmed segments, and the information of the corresponding unconfirmed segment is deleted. Here, the threshold value d may have an appropriate value in advance, for example, the maximum character width w or character size as a predetermined value, or when the handwritten character input is a single line input, the frame height of the single line input may be used, and is not particularly limited. Also, the threshold value d may be calculated, for example, as the maximum character width w×n, instead of using the maximum character width w, character size, or frame height of the single line input itself. Usually, n is in the range of 1.0 to 2.0, but is not particularly limited.

[0111] The handwritten character detection unit 60 creates a set of confirmed segments that can be one character from the series of confirmed segments as character region candidates, and creates a network in which each character region is connected by a link. The handwritten character detection unit 60 then determines a character region by combining multiple adjacent segments. FIG. 16A shows an example of a combination of segments in the "Bunka na" part when "Bunka no recognition" is handwritten. Links are established between individual character regions that can be adjacent. FIG. 16B shows possible combination links with arrows when "Bunka no" is handwritten. For example, the first "木" and "卆" can be connected, so a link is established from "卆" to "木", but "木" and "な" are not adjacent character regions because there is "卆" between them, so they cannot be connected and no link is established.

[0112] If adjacent segments are combined, a large number of combinations are possible, but the combinations must be plausible as a single character, so the number of combinations is quite limited. In Figure 16B, the criterion used is that only segments whose x-direction width of the character area is within 1.5 times the standard character size are combined, but this number is not a limit.

[0113] The character detection method may be a machine learning model that detects characters from drawing data.

[0114] Returning to Fig. 14, the handwritten character detection unit 60 calculates the average size of the extracted characters. The character size is expressed in points corresponding to the size of the extracted characters. It may also be expressed in millimeters.

[0115] The drawing data zoom unit 61 calculates the ratio between the average character size of group 1 and the average character size of group 2, and if the ratio exceeds a threshold value (e.g., 1.5), it adjusts the size of the drawing data of group 1 (an example of the first group) and group 2 (an example of the second group) so that the ratio between the two becomes the threshold value.

[0116] When the average of the two font sizes in Group 1 is 120 pt and the average of the three font sizes in Group 2 is 30 pt, if the reduction rate of Group 1 is x and the enlargement rate of Group 2 is 1 / x, the calculation formula for making the ratio of the two equal to the threshold is as follows. (120×x) / (30×1 / x) = 1.5, x ≒ 0.612 Therefore, the drawing data enlargement / reduction unit 61 of the conference server 1 reduces the drawing data of Group 1 by 0.612 times and enlarges the drawing data of Group 2 by 1 / 0.612 times to make the ratio of the average of both font sizes equal to the threshold. The difference in font size between Groups 1 and 2 is reduced.

[0117] Group 1 contains the square 205 and the star mark 204, and the drawing data enlargement / reduction unit 61 also reduces these figures by 0.612 times. Group 2 contains the arrow 206, and the drawing data enlargement / reduction unit 61 also enlarges this arrow 206 by 1 / 0.612 times.

[0118] Therefore, Group 1 is fixed at the coordinates (0,0) and reduced by 0.612 times, and Group 2 is fixed at the coordinates (x3,y3) and enlarged by 1 / 0.612 times.

[0119] Figure 17 shows the drawing data in which Group 1 is reduced and Group 2 is enlarged. The two handwritten data 203 of "あ", the square 205, and the star mark 204 are reduced, and the three handwritten data 208 of "い" and the arrow 206 are enlarged, and the size difference between Group 1 and Group 2 is reduced.

[0120] The drawing data transmission unit 57 of the conference server 1 transmits the enlarged or reduced drawing data to the electronic blackboard 2. The drawing data reception unit 42 of the electronic blackboard 2 at Site 1 receives the enlarged or reduced drawing data, and the display control unit 44 causes it to be displayed in the display area 201. Therefore, the drawing data of all devices is clearly displayed in the display area 201 of the electronic blackboard 2.

[0121] <Operation or process> FIG. 18 is a sequence diagram for explaining the process in which the conference server 1 moves the drawing data of the tablet PC 3 and displays it in the display area of the electronic blackboard 2. In the description of FIG. 18, the differences from FIG. 12 will mainly be explained.

[0122] In the sequence diagram of FIG. 18, step S41 is added. S41: When the transfer of the drawing data of the group is completed, the handwritten character detection unit 60 of the conference server 1 detects handwritten characters from each group and calculates the average size of the characters cut out for each group. The drawing data enlargement / reduction unit 61 obtains the ratio of the average size of the characters in group 1 to the average size of the characters in group 2, and when the ratio exceeds a threshold value (for example, 1.5), adjusts the sizes of the drawing data of group 1 and group 2 so that the ratio becomes the threshold value.

[0123] <Main effects> According to the present embodiment, by reducing the drawing data, it becomes easier to display a list of the drawing data of all bases. When it is desired to check the drawing data of all devices, the user can save the trouble of scrolling the screen, and the conference time can be shortened.

[0124] [Third Embodiment] In the present embodiment, the conference server 1 that moves the drawing data of one group to the empty space in the drawable area 217 will be described, rather than simply bringing two groups closer.

[0125] Note that in the present embodiment, the functional block diagram of FIG. 13 described in the second embodiment can be used for the description.

[0126] <Flow of processing by the conference server> FIG. 19 shows the handwritten data 203 drawn on the electronic blackboard 2 at Site 1. At the electronic blackboard 2 at Site 1, when a user writes by hand, the contact position detection unit 45 detects the contact position, and the display control unit 44 generates and displays handwritten data at that position. The drawing data transmission unit 41 transmits the drawing data related to the handwritten data to the conference server 1. When the conference server 1 receives the drawing data from Site 1, it transmits the drawing data to the tablet PC 3 at Site 2. The tablet PC 3 at Site 2 receives and displays the drawing data.

[0127] The user at Site 2 wants to add a note in the empty space below the handwritten data 203 at Site 1. However, since there is little empty space, the displayable area 217 is scrolled upward (the display area is scrolled downward) to expand the empty space. The contact position detection unit 51 detects the upward scroll, and the display control unit 50 moves the display area.

[0128] FIG. 20 shows the display area 202 drawn by the tablet PC 3 at Site 2. The user at Site 2 handwrote 18 handwritten data 225 (characters) of "i". When the user writes by hand on the tablet PC 3 at Site 2, the contact position detection unit 51 detects the contact position, and the display control unit 50 generates and displays handwritten data at that position. The drawing data transmission unit 47 of the tablet PC 3 transmits the drawing data related to the handwritten data to the conference server 1.

[0129] However, when the handwritten data 225 is drawn as shown in FIG. 20, since the range of the handwritten data at Site 2 is wide, it will protrude from the display area 201 at Site 1. Alternatively, when the electronic blackboard 2 is reduced in size, the characters will become small.

[0130] FIG. 21 shows the handwritten data of the electronic blackboard 2 and the handwritten data of the tablet PC 3 drawn in the drawable area 217 of the conference server 1. When attempting to display such handwritten data on the electronic blackboard 2, the handwritten data written by the user at site 2 protrudes from the display area 201 of the electronic blackboard 2. In such a situation, simply moving the drawing data of group 2 closer to the drawing data of group 1 as in the first embodiment cannot eliminate the protrusion of the handwritten data written by the user at site 2.

[0131] Therefore, in this embodiment, the conference server 1 adjusts the position of the handwritten data at site 2. The conference server 1 holds the number of display pixels in the display areas of the devices at each site. According to the aspect ratio, the number of display pixels of the electronic blackboard 2 is landscape. Therefore, if the drawing data of the tablet PC 3 is moved to the right of the handwritten data 203, it is possible to prevent the characters from becoming small even when the aspect ratio is maintained and reduced.

[0132] Specifically, assume that the display area of the first device that requested the list display of the drawing data is landscape, and in the drawable area, drawing data is written by the second device below the drawing data drawn in the display area of the first device, and the drawing data of the first device and the second device does not fit in the display area of the first device. In this case, the drawing data moving unit 56 moves the second group of drawing data drawn by the second device horizontally to the first group of drawing data drawn by the first device.

[0133] FIG. 22 is a diagram for explaining a method of adjusting the position of the handwritten data at site 2 by the conference server 1. When the user at site 1 presses the overall display button 210 to view all the drawing data on the whiteboard (the memory of the conference server 1) on the electronic blackboard 2, the electronic blackboard 2 sends an overall display command to the conference server 1.

[0134] When the conference server 1 receives the overall display command, the drawing data grouping unit 54 of the conference server 1 groups the drawing data of each site. The grouping method may be specified by the circumscribed rectangles 226 and 227 in the same manner as in the first embodiment. Then, the drawing data moving unit 56 and the drawing data enlarging / shrinking unit 61 adjust the position and size of the drawing data of the tablet PC 3 (an example of the second device) at site 2 so as to enter the display area 201 of the electronic blackboard 2 (an example of the first device) at site 1. The drawing data moving unit 56 compares the reduction ratios when reducing after moving as in the first embodiment so as to enter the display area 201, and when reducing after moving group 2 to the right side of group 1 so as to enter the display area 201. Here, it is assumed that the reduction ratio is smaller (the characters do not become smaller) in the latter case.

[0135] Figure 23 shows the circumscribed rectangle 229 circumscribing each group so as to include each group. Two groups 1 and 2 are specified by the rectangular area surrounded by the circumscribed rectangle 229. The coordinates of the upper left corner of this circumscribed rectangle 229 are set to (0, 0). The coordinates of the upper right corner of group 1 are set to (x1, 0), and the coordinates of the upper left corner of group 2 are set to (0, y2). The drawing data moving unit 56 of the conference server 1 moves the drawing data of group 2 to the right side of group 1. For this purpose, the drawing data moving unit 56 of the conference server 1 moves group 2 horizontally (to the right) by x1 + α and vertically upward by y2 so that the position obtained by adding a margin of α to the coordinates (x1, 0) of the upper right corner of group 1 becomes the upper left corner of group 2.

[0136] Figure 24 shows the drawable area 217 in which the drawing data of group 2 has been moved to the right side of group 1. Next, the drawing data enlarging / shrinking unit 61 determines whether the size of the entire groups 1 and 2 is less than or equal to the size of the display area 201 of the electronic blackboard 2 as shown in Figure 24. If the size of the entire groups 1 and 2 exceeds the size of the display area 201 of the electronic blackboard 2, the drawing data enlarging / shrinking unit 61 adjusts so that the entire groups 1 and 2 fit within the display area 201 of the electronic blackboard 2.

[0137] That is, when the drawing data enlargement / reduction unit 61 cannot fit the entire drawing data of the first group and the moved second group in the display area of the first device, it reduces the size of the entire drawing data so that it can fit in the display area of the electronic blackboard 2 that requested the list display of the drawing data.

[0138] FIG. 25 is an example of a diagram showing the relationship between the drawing data before reduction and the display area 201, and the relationship between the drawing data after reduction and the display area 201. As shown in FIG. 25(a), the drawing data of base point 1 and the drawing data of base point 2 after movement are specified by the circumscribed rectangle 228. FIG. 25(a) shows the drawing data before reduction, and the size of the entire groups 1 and 2 is larger than the display area 201 of the electronic blackboard 2. Assume that the size of the entire groups 1 and 2 is 2500×800, and the size of the display area 201 of base point 1 is 1920×1080. In this case, if the reduction rate of the entire groups 1 and 2 is x, the calculation formula for the reduction rate x at which the entire groups 1 and 2 can fit in the display area 201 of the electronic blackboard 2 is as follows. That is, the ratio of the height and width of the display area 201 and the entire groups 1 and 2 is obtained, and the smaller one is adopted as the reduction rate x. x = Min (1920 / 2500, 1080 / 800) ≒ Min (0.768, 1.35) = 0.768 Therefore, the drawing data enlargement / reduction unit 61 of the conference server 1 reduces the size of the entire groups 1 and 2 to 0.768 times while maintaining the aspect ratio.

[0139] FIG. 25(b) shows the drawing data of groups 1 and 2 after adjustment (after reduction), and the entire drawing data of groups 1 and 2 fits in the display area 201 of the electronic blackboard 2. Note that FIG. 25 shows an example of reduction, but when x is larger than 1 according to the same calculation formula, the drawing data can also be enlarged. That is, the electronic blackboard 2 can display the drawing data of groups 1 and 2 using the entire display area 201.

[0140] The drawing data transmission unit 57 of the conference server 1 transmits the drawing data of the entire Groups 1 and 2 after enlargement or reduction to the electronic blackboard 2. The drawing data reception unit 42 of the electronic blackboard 2 at Site 1 receives the drawing data, and the display control unit 44 causes the drawing data to be displayed.

[0141] <Operation or process> FIG. 26 is a sequence diagram for explaining the process in which the conference server 1 moves the drawing data of the tablet PC 3 and causes it to be displayed in the display area of the electronic blackboard 2. In the description of FIG. 26, the differences from FIG. 12 will mainly be explained.

[0142] In the sequence diagram of FIG. 26, steps S42 and S43 are added. S42: The drawing data movement unit 56 compares the reduction ratios when simply approaching and when moving so as to be aligned horizontally, and selects the movement method with the smaller reduction ratio. For example, the drawing data movement unit 56 fixes Group 1 of the drawing data on the electronic blackboard 2 and moves Group 2 of the drawing data on the tablet PC 3 to the right side of this Group 1. As described above, the drawing data movement unit 56 calculates the movement amount so that the upper right corner of Group 1 and the upper left corner of Group 2 substantially coincide, and moves Group 2 in the horizontal and vertical directions.

[0143] S43: Next, the drawing data enlargement / reduction unit 61 obtains the reduction ratio x so that the drawing data of all groups fits within the display area 201 of the electronic blackboard 2, and reduces it at the reduction ratio x while maintaining the aspect ratio of the entire drawing data. Note that the drawing data enlargement / reduction unit 61 may reduce or enlarge each group of the drawing data at each site, instead of reducing the entire drawing data, as in the first embodiment.

[0144] <Main effects> According to the present embodiment, since the drawing data is moved in units of groups and the drawing data can be reduced at the minimum reduction ratio so as to fit within the display area of the device, the drawing data of all sites can be displayed in a list. When it is desired to check the drawing data of all devices, the user can be spared the trouble of scrolling the screen, and the conference time can be shortened.

[0145] [Embodiment 4] Users at each site may scroll the display area and write by hand in the blank space. When a user at an arbitrary site views an overview of all the drawing data, the conference server 1 adjusts the position and size of the drawing data so that the drawing data from each site is displayed in a well-balanced manner. In this embodiment, the conference server 1 that adjusts the size and position of the drawing data so that the drawing data from the four sites does not overlap with the drawing data from other sites will be described. <Regarding the functions> The functional block diagram of the conference server 1 in this embodiment is the same as that in the second embodiment (FIG. 13), and the functional block diagrams of the electronic blackboard 2 and the tablet PC 3 are the same as those in the first embodiment (FIG. 5).

[0146] <Drawing data displayed by devices at four sites> An overview of the processing flow described with reference to FIGS. 27 to 34 will be given. The drawing data receiving unit 53 of the conference server 1 receives drawing data from N (4 in the figure, but 4 is just an example) electronic blackboards 2. The drawing data grouping processing unit groups the drawing data from the four electronic blackboards 2 written in the drawable area for each device. The handwritten character detection unit 60 detects characters from the drawing data of each group. The drawing data enlargement / reduction unit 61 in this embodiment calculates a first average of the character sizes for each group, calculates a second average of the character sizes between the groups based on the first average, and enlarges or reduces the drawing data of each group by the ratio of the second average to the first average. The drawing data moving unit 56 evenly divides the display area of the electronic blackboard 2 that requested the list display of the drawing data into four areas, and arranges the enlarged or reduced drawing data of each group in each area.

[0147] Figure 27 shows the drawing data displayed by the devices at four bases. Note that the device at Base 1 is the electronic blackboard 2, the device at Base 2 is the tablet PC 3, and the devices at Bases 3 and 4 are the electronic blackboard 2. Figure 27(a) shows the handwritten data 203 and the square 205 drawn at Base 1, Figure 27(b) shows the handwritten data 225 drawn at Base 2, Figure 27(c) shows the handwritten data 231 drawn at Base 3, and Figure 27(d) shows the handwritten data 232 and the star mark 233 drawn at Base 4.

[0148] For example, a user at Base 1 presses the overall display button 210 to view the drawing data for the entire area of each base. When the electronic blackboard 2 detects the pressing of the overall display button 210, it sends an overall display command to the conference server 1. When the conference server 1 receives the overall display command, the drawing data grouping unit 54 of the conference server 1 groups the drawing data of each base respectively.

[0149] Figure 28 shows the drawing data of four bases 1 to 4 grouped by circumscribed rectangles 235 to 238 in the drawable area 217. The drawing data of Bases 1 to 4 are respectively set as Groups 1 to 4. The handwritten character detection unit 60 of the conference server 1 cuts out the characters within each of Groups 1 to 4. The cutting method may be the same as that in the second embodiment.

[0150] Figure 29 schematically shows the characters cut out from Groups 1 to 4. 'A', 'I', 'U', and 'E' are respectively detected from Groups 1 to 4. The handwritten character detection unit 60 calculates the average size of the characters for each group. The size of the characters is represented by the number of points corresponding to the size of the cut-out characters. It may also be represented in millimeters.

[0151] As shown in Figure 29, the drawing data enlargement / reduction unit 61 calculates the average (an example of the first average) of the sizes of the characters cut out from the drawing data of Bases 1 to 4 for each group. · The drawing data enlargement / reduction unit 61 cuts out two characters at Base 1 and obtains the average of the number of points corresponding to the size of each character. Let the average of the characters at Base 1 be a1. · The drawing data enlargement / reduction unit 61 extracts 18 characters of base point 2 and obtains the average of the number of points corresponding to the size of each character. Let the average of the characters of base point 2 be a2. · The drawing data enlargement / reduction unit 61 extracts 3 characters of base point 3 and obtains the average of the number of points corresponding to the size of each character. Let the average of the characters of base point 3 be a3. · The drawing data enlargement / reduction unit 61 extracts 4 characters of base point 4 and obtains the average of the number of points corresponding to the size of each character. Let the average of the characters of base point 4 be a4.

[0152] The average W of the averages a1 to a4 of the sizes of the characters of base points 1 to 4 is W = (a1 + a2 + a3 + a4) / 4.

[0153] The drawing data enlargement / reduction unit 61 of the conference server 1 uses the average W (an example of the second average) to enlarge or reduce the drawing data of each base point as follows. · Since the average (a1) of the sizes of the characters of base point 1 is larger than W, all drawing data (including figures) is reduced by W / a1 (reduced for each group). · Since the average (a2) of the sizes of the characters of base point 2 is smaller than W, all drawing data is enlarged by W / a2 (enlarged for each group). · Since the average (a3) of the sizes of the characters of base point 3 is larger than W, all drawing data is reduced by W / a3 (reduced for each group). · Since the average (a4) of the sizes of the characters of base point 4 is larger than W, all drawing data (including figures) is reduced by W / a4 (reduced for each group).

[0154] Figure 30 shows the drawing data after enlargement or reduction. It can be seen that the drawing data of base points 1, 3, and 4 has become smaller, and the drawing data of base point 2 has become larger.

[0155] Next, the drawing data movement unit 56 of the conference server 1 evenly divides the display area 201 of the electronic blackboard 2 of base point 1 into 4 areas, which is the number of areas of base points 1 to 4. That is, the drawing data movement unit 56 divides the display area 201 into the same number as the number of base points.

[0156] FIG. 31 shows a drawable area 217 in which the display area 201 of the electronic blackboard 2 at base 1 is equally divided into four. An overview will be described. The drawing data moving unit 56 arranges the drawing data of each group enlarged or reduced so that the upper left corners of each group coincide with the upper left corners of the four areas. When the first drawing data protrudes from the area assigned to an arbitrary group to another area, the drawing data moving unit 56 moves the second drawing data in another area by the amount that the first drawing data protrudes in the direction in which the first drawing data protrudes in the other area.

[0157] Note that the correspondence between areas 1 to 4 and bases 1 to 4 may be determined in advance, but it is preferable that the relative positional relationship of the handwritten drawing data is maintained. For this reason, the drawing data moving unit 56 calculates the distance between the center of gravity of the drawing data of each base and the center of each area, and associates the drawing data of each base with the area having the shortest distance. In FIG. 31, as an example, it is assumed that base 1 corresponds to area 1, base 2 corresponds to area 2, base 3 corresponds to area 3, and base 4 corresponds to area 4.

[0158] This will be described in detail below. The drawing data moving unit 56 determines whether the drawing data of the base corresponding to each area fits.

[0159] First, the movement with the origin aligned will be described. When the drawing data in the enlarged or reduced group fits in the divided area, the drawing data moving unit 56 moves to the upper left of that area so that the upper left corner of the group coincides. When the drawing data in the enlarged or reduced group does not fit in the divided area, the drawing data in the area in the protruding direction is moved by the amount that it protrudes in the protruding direction. In FIG. 31, the drawing data of groups 1, 3, and 4 fits in areas 1, 3, and 4, but the drawing data of group 2 does not fit in area 2.

[0160] The drawing data moving unit 56 calculates the distances to move in the X-axis direction and the Y-axis direction from the differences between the upper left coordinates of each of the regions 1 to 4 and the upper left coordinates of each group, and moves each group in order. The drawing data moving unit 56 moves, for example, from the region 1 at the uppermost left in the order of the upper right region 2, the lower left region 3, and the lower right region 4. Note that the following moving distances are absolute values. · Group 1 is moved (0 - x1) in the left direction of the X-axis and (0 - y1) in the upward direction of the Y-axis. · Group 2 is moved (x - x2) in the left direction of the X-axis and (0 - y2) in the downward direction of the Y-axis, and is in a state of protruding downward from region 2. · Group 3 is moved (0 - x3) in the left direction of the X-axis and (y - y3) in the upward direction of the Y-axis. · Since group 2 protrudes into region 4, group 4 is moved below the moved group 2. Therefore, group 4 is moved (x - x4) in the left direction of the X-axis and (y4 - (y2 + y2')) in the upward direction of the Y-axis.

[0161] FIG. 32 shows the drawing data of each of the groups 1 to 4 after the movement. Groups 1 and 3 that fit within regions 1 and 3 align the upper left corner with the upper left corner of the region, and group 2 that does not fit within the region is moved by the amount that it protrudes beyond group 4 in the region in the protruding direction.

[0162] Next, the horizontal movement will be described. For the vertical direction, the upper left corners of regions 1 to 4 and the upper left corners of the groups were made to coincide as much as possible, or the groups were moved so that there was no gap between the groups. However, the drawing data of group 2 protrudes to the right from the display area 201 of the electronic blackboard 2. Also, there is an empty space to the right of group 1 or 3. Therefore, the drawing data moving unit 56 determines whether there is an empty space between group 2 and groups 1 and 3 on the left side. As shown in FIG. 32, there is an empty space of w1 = x - x5 between group 2 and group 1, and an empty space of w2 = x - x6 between group 3 and group 4.

[0163] In the case of w1 and w2, since the blank space of w1 is narrower, the drawing data moving unit 56 moves group 2 only by w1 in the negative X-axis direction (left direction) up to the vicinity of group 1.

[0164] FIG. 33 shows a drawable area 217 in which group 2 has been moved leftward by w1 up to the vicinity of group 1. That is, when the group assigned to the right area by dividing the display area 201 of the electronic blackboard 2 into four parts does not fit horizontally in the display area 201 of the electronic blackboard 2 that has requested a list display of drawing data, and there is a blank space between the group in the right area and the group in the left area, the drawing data moving unit 56 moves the right group leftward so that there is no blank space.

[0165] Note that since the drawing data of group 4 fits within the X-axis direction of the display area of the electronic blackboard 2 (i.e., it does not protrude), it is not moved in the X-axis direction.

[0166] FIG. 34 shows all the drawing data list-displayed by the electronic blackboard 2. It can be seen that all the drawing data fits within the display area 201 of the electronic blackboard 2.

[0167] <Operation or Process> FIG. 35 is a sequence diagram for explaining the process in which the conference server 1 list-displays the drawing data of four bases. Note that in the description of FIG. 35, the differences from FIG. 12 will mainly be explained.

[0168] In the sequence diagram of FIG. 35, steps S51 to S53 are added. S51: The handwritten character detection unit 60 of the conference server 1 cuts out the characters in each of groups 1 to 4 in the drawable area 217. Also, the handwritten character detection unit 60 calculates the average size of the characters for each group.

[0169] S52: The drawing data enlargement / reduction unit 61 of the conference server 1 calculates the average W of the average sizes a1 to a4 of the characters at bases 1 to 4. Then, the drawing data enlargement / reduction unit 61 enlarges or reduces the drawing data for each base using the average W.

[0170] [S53: Next, the drawing data enlargement / reduction unit 61 evenly divides the display area 201 of the electronic blackboard 2 where the whole display button 210 is pressed into four parts. The drawing data movement unit 56 moves the group so that the upper left corner of the area coincides with the upper left corner of the group. When moving, if there is drawing data that protrudes into other areas, the drawing data in the protruding destination area is moved by the amount of protrusion. Further, when the right group protrudes from the display area 201 in the X-axis direction (horizontal direction) and there is a vacant space in the X-axis direction between the left group and the right group, the drawing data movement unit 56 moves the group to the left so that the narrower vacant space disappears.

[0171] <Main effects> [According to this embodiment, when there are four bases, by enlarging or reducing the size of the characters and moving the group to the areas obtained by dividing the display area into four parts, all the drawing data of each base can be displayed in a list. Note that the number of bases is not limited to four.

[0172] [Fifth Embodiment] [In this embodiment, a case will be described where a user at base 2 (home) participates in a virtual meeting using a VR goggles and a VR operation controller, and a user at base 1 (meeting room) participates in this meeting using an electronic blackboard 7.

[0173] Similarly, in a virtual meeting, the meeting server can also display a list of the drawing data of each site. Specifically, the virtual meeting service executed on the meeting server transmits the drawing data displayed in the virtual space to the electronic blackboard 7, and also displays the drawing data received from the electronic blackboard 7 in the virtual space. The drawing data transmitted from the electronic blackboard 7 is written into the drawable area on the memory of the meeting server, or the drawing data transmitted from the electronic blackboard 7 is written into any of a plurality of pages. When a request for a list display of the drawing data is made from the electronic blackboard 7, the drawing data moving unit 56 moves at least a part of the drawing data in the drawable area to the area corresponding to the display area and arranges the drawing data of a plurality of sites in the area corresponding to the display area, or arranges the drawing data of a plurality of pages in the area corresponding to the display area. The drawing data transmitting unit 57 transmits the drawing data of each site arranged by the drawing data moving unit 56 in the area corresponding to the display area to the electronic blackboard 7. This will be described in detail below.

[0174] <System configuration example> FIG. 36 shows a system configuration diagram of a virtual meeting system 700. An electronic blackboard 7 is arranged in a meeting room (site 1) where users A to C are present, and user D wearing a VR goggle 10 is at home. A meeting server 4, a virtual meeting server 5, and a meeting reservation server 6 are connected to a communication network N such as the Internet.

[0175] The meeting server 4 corresponds to the meeting server 1 described in the first to fourth embodiments, and a function of cooperating with the virtual meeting server 5 is added. The virtual meeting server 5 stores the data of the virtual meeting space 18 in the memory and executes the virtual meeting application. Hereinafter, the virtual meeting application executed on the virtual meeting server 5 is referred to as a virtual meeting service. The virtual meeting space 18 is, for example, a 360° three-dimensional space that imitates a meeting room in a virtual space.

[0176] The data of the virtual conference space 18 is the screen data of each device. In the data of the virtual conference space 18, the screen data of the electronic blackboard 7 is called the virtual whiteboard 18a. In the data of the virtual conference space 18, the screen data of the user D's notebook PC 17 is called the virtual PC screen data 18b. The virtual whiteboard 18a is synchronized with the display data of the electronic blackboard 7, and the virtual PC screen data 18b is synchronized with the screen data of the PC.

[0177] The conference reservation server 6 holds the conference ID, scheduled date and time, participants, etc. of the conference, which are generated for each conference, for the conferences executed by the conference server 4, the virtual conference server 5, or the conferences executed in cooperation with both of them.

[0178] An electronic blackboard 7 is installed in the conference room where users A, B, and C are. The electronic blackboard 7 is connected to the communication network N and can communicate with the conference server 4, the virtual conference server 5, and the conference reservation server 6.

[0179] User D is at home (or it can also be a satellite office, etc.), and a Wi-Fi router 8 is installed at home. User D is using a notebook PC 17, VR goggles 10, and a VR operation controller 11. The Wi-Fi router 8 at user D's home can be connected to the communication network N. The notebook PC 17 and the VR goggles 10 of user D can communicate wirelessly with the Wi-Fi router 8. The VR goggles 10 and the VR operation controller 11 can communicate wirelessly, such as via Bluetooth. Note that the notebook PC 17 and the VR goggles 10 are also connected via Bluetooth, etc. A virtual conference app that communicates with the virtual conference server 5 is launched on the notebook PC 17 of user D. There may be multiple users who participate in the conference while wearing the VR goggles 10.

[0180] The drawing data handwritten on the electronic blackboard 7 at Site 1 is transmitted to the conference server 4. The conference server 4 writes the drawing data into the drawable area 217 and transmits it to the virtual conference server 5. The virtual conference server 5 writes the drawing data into the virtual whiteboard 18a and transmits the video of the virtual conference space 18 to the VR goggles 10. As a result, the drawing data of the electronic blackboard 7 is synchronized between Site 1 and Site 2. Similarly, the drawing data handwritten by the user on the virtual whiteboard 18a in the virtual conference space 18 at Site 2 is transmitted to the virtual conference server 5. The virtual conference server 5 writes the drawing data into the virtual whiteboard 18a and transmits it to the conference server 4. The conference server 4 writes the drawing data into the drawable area 217 and transmits it to the electronic blackboard 7. As a result, the drawing data of the virtual whiteboard 18a is synchronized between Site 2 and Site 1.

[0181] <Hardware Configuration Example> <<VR Goggles>> Figure 37 is a hardware configuration diagram of the VR goggles 10. The VR goggles 10 include a CPU 80, a main memory 81, a ROM 82, a display controller 86, a wireless LAN controller 88, an audio codec 90, and a moving image codec 93 connected via a bus 94.

[0182] The CPU 80 executes and processes the OS and control processing programs read from the ROM 82 into the main memory 81. The main memory 81 has a DRAM and is used as a work area of the CPU 80 and the like.

[0183] The ROM 82 has an OS, a system program at power-on, and a program for controlling the VR goggles 10 pre-written therein.

[0184] A UART 83 is connected to the CPU 80. The UART 83 is an interface for exchanging serial data between the CPU 80 and the Bluetooth module 84, and is composed of a FIFO (First In, First Out), a shift register, and the like.

[0185] The Bluetooth module 84 is composed of an RF section and a baseband section, and an antenna 85 is connected thereto, and it performs wireless communication compliant with the Bluetooth standard.

[0186] The display controller 86 performs D / A conversion on character data, graphic data, image data, etc., and controls the display of these data on the LCD 87.

[0187] The wireless LAN controller 88 executes a communication protocol compliant with the IEEE (Institute of Electrical and Electronics Engineers) 802.11ax standard, and controls communication with other devices by transmitting and receiving radio waves via the antenna 89.

[0188] The voice signal input from the microphone 91 is converted into voice data by an A / D (Analog / Digital) conversion circuit, and this voice data is encoded by the voice codec 90 in the AAC (Advanced Audio Coding) format. Also, the AAC encoded data received from an external device is decoded by the voice codec 90, and the voice converted into analog by the D / A conversion circuit is output from the speaker 92. The moving image codec 93 decodes the moving image compression data received from an external device (for example, a format compliant with ITU-T Recommendation H.264). Data is transferred between the above-mentioned respective components via the bus 94.

[0189] <<VR Operation Controller>> Figure 38 is a hardware configuration diagram of the VR operation controller 11. The VR operation controller 11 has a UART 117, a main memory 111, a ROM 112, a 6-axis acceleration / angular velocity sensor 113, a menu display button 114, a pointer display button 115, and an enter button 116 connected to the CPU 110.

[0190] The CPU 110 executes and processes the control processing program read from the ROM 112 into the main memory 111. The main memory 111 has a DRAM and is used as a work area etc. of the CPU 110.

[0191] In the ROM 112, a system program at power-on, a program for transmitting the press information of the menu display button 114, the pointer display button 115, and the confirmation button 116 via Bluetooth is pre-written.

[0192] The 6-axis acceleration / angular velocity sensor 113 outputs measurement data of acceleration and angular velocity. The UART 117 is an interface for exchanging serial data between the CPU 110 and the Bluetooth module 118, and is composed of a FIFO, a shift register, etc. The Bluetooth module 118 is composed of an RF section and a baseband section, and an antenna 119 is connected thereto, and performs wireless communication compliant with the Bluetooth standard.

[0193] <Regarding functions> <<Conference server and electronic blackboard>> FIG. 39 is a functional block diagram for explaining the functions of the conference server 4 and the electronic blackboard 7 in blocks. In the explanation of FIG. 39, the differences from FIG. 5 of the first embodiment are mainly explained. The conference server 4 in FIG. 39 newly has a virtual conference connection section 63 and a virtual whiteboard synchronization control section 64.

[0194] The virtual conference connection section 63 connects to the virtual conference service of the virtual conference server 5 and establishes a data transfer session between the conference server 4 and the virtual conference server 5. That is, for the conferences registered in the conference reservation server 6, the virtual conference connection section 63 authenticates the registered devices and then establishes a data transfer session between the devices and the virtual room (managed by the virtual conference server 5 and participated by user D).

[0195] The virtual whiteboard synchronization control unit 64 synchronizes the drawing data of the electronic blackboard 7 (the drawing data of the drawable area 217) on the memory with the drawing data of the virtual whiteboard 18a of the virtual conference server 5 (maintains the data of both to be the same).

[0196] <<Electronic blackboard>> The electronic blackboard 7 includes a contact position detection unit 70, a display control unit 71, a LAN communication control unit 72, a Bluetooth communication control unit 73, a device information transmission unit 46, a drawing data generation unit 74, a virtual conference connection unit 75, a data storage unit 76, a drawing data transmission unit 41, and a drawing data reception unit 42. Each of these units included in the electronic blackboard 7 is a function or means realized by any of the components shown in FIG. 3 operating according to an instruction from the CPU 401 according to a program (virtual conference application) expanded from the SSD 404 to the RAM 403. Note that the functions of the drawing data transmission unit 41 and the drawing data reception unit 42 may be the same as those in the first embodiment.

[0197] The contact position detection unit 70 detects the coordinate data of the portion (the portion where light is blocked) detected by the contact sensor 414 when a finger, an electronic pen 490, etc. touches the display 480.

[0198] The display control unit 71 displays the data of the virtual whiteboard 18a received by the LAN communication control unit 72 on the display 480. Specifically, the display control unit 71 performs control for displaying the screen data of the electronic blackboard 7 on the display 480.

[0199] The LAN communication control unit 72 is a means for transmitting and receiving data other than the drawing data to and from the conference server 4. Specifically, the LAN communication control unit 72 connects to a communication network N such as Ethernet (registered trademark) and transmits and receives data to and from other devices via the communication network N. The LAN communication control unit 72 may communicate with the virtual conference server 5.

[0200] The Bluetooth communication control unit 73 executes communication compliant with the Bluetooth standard. The drawing data generation unit 74 generates drawing data from the sequence of coordinate points input from the contact position detection unit 70 when the user writes by hand, or generates figures such as circles and rectangles.

[0201] The virtual conference connection unit 75 performs control to connect to the virtual conference service executed on the virtual conference server 5 using the URL and conference ID of the virtual conference service transmitted from the notebook PC 17. In this embodiment, since the electronic blackboard 7 is connected to the virtual conference server 5 via the conference server 4, the virtual conference connection unit 75 is not used.

[0202] The data storage unit 76 stores the address (URL) and conference ID of the virtual conference service provided by the virtual conference server 5, and the screen data displayed on the display 480, etc. The device information transmission unit 46 transmits device information to the conference server 4.

[0203] <<Virtual Conference Server>> FIG. 40 is an example of a functional block diagram for explaining the functions of the virtual conference server 5, the notebook PC 17, the VR goggles 10, and the VR operation controller 11 by dividing them into blocks.

[0204] The virtual conference server 5 includes a virtual conference control unit 20, a whiteboard generation and management unit 21, a synchronization control unit 22, a communication control unit 23, a conference data storage unit 24, a display data generation unit 25, a virtual display control unit 26, a pointer position calculation unit 27, a user authentication unit 28, and a user information storage unit 29. Each of these units included in the virtual conference server 5 is a function or means realized by the operation of any of the components shown in FIG. 4A according to an instruction from the CPU 501 in which a program expanded from the HD 504 to the RAM 503.

[0205] The virtual conference control unit 20 associates the participants of the conference and the devices connected to the conference (laptop PC 17, VR goggles 10, conference server 4) with the identification information of the conference (conference ID) and stores them in the conference data storage unit 24. The virtual conference control unit 20 executes a remote conference with each device for each conference ID. Further, the virtual conference control unit 20 stores the screen data of the virtual whiteboard and the audio data received from the devices connected to the virtual conference in the conference data storage unit 24.

[0206] The whiteboard generation and management unit 21 generates and manages the virtual whiteboard 18a and the virtual PC screen data 18b in the virtual conference space 18.

[0207] The synchronization control unit 22 is a means for synchronizing the screen data of the electronic blackboard 7 on the memory of the conference server 4 and the virtual whiteboard 18a so that they are the same. Specifically, the synchronization control unit 22 includes a function of detecting the update of the screen data of the electronic blackboard 7 on the memory of the conference server 4 and the virtual whiteboard 18a, a function of reflecting the updated screen data of the electronic blackboard 7 on the memory of the conference server 4 on the virtual whiteboard 18a, and a function of reflecting the updated virtual whiteboard 18a on the screen data of the electronic blackboard 7 on the memory of the conference server 4.

[0208] The communication control unit 23 executes data transmission and reception with the laptop PC 17, VR goggles 10, and conference server 4.

[0209] The conference data storage unit 24 stores the virtual whiteboard 18a, the virtual PC screen data 18b, and the audio data received from the devices connected to the virtual conference.

[0210] The display data generation unit 25 generates the menu displayed in the virtual conference space 18, the pointer indicated by the display instruction by the VR operation controller 11, and the like.

[0211] The virtual display control unit 26 is a means for displaying drawing data, screen data of the notebook PC 17, etc. based on the operation information transmitted from the VR goggles 10 in the virtual conference space 18. Specifically, the virtual display control unit 26 displays various display data in the virtual conference space 18. The virtual display control unit 26 displays a menu generated by the display data generation unit 25, a pointer, a virtual whiteboard 18a generated by the whiteboard generation and management unit 21, virtual PC screen data 18b of the notebook PC 17, etc. in the virtual conference space 18. The virtual display control unit 26 transmits the display data of the virtual conference space 18 to the VR goggles 10 via the communication control unit 23, and the virtual conference space 18 is displayed on the VR goggles 10.

[0212] The pointer position calculation unit 27 calculates the position of the pointer (in the virtual conference space 18) indicated by the orientation and movement of the VR operation controller 11.

[0213] The user authentication unit 28 collates the user name and password received via the communication control unit 23 with the user name and password stored in the user information storage unit 29 to authenticate the user participating in the virtual conference.

[0214] The user information storage unit 29 stores the user name and password of the user participating in the virtual conference.

[0215] <<Notebook PC>> The notebook PC 17 has a virtual conference connection unit 30, a remote desktop control unit 31, a wireless LAN communication control unit 32, a Bluetooth communication control unit 33, a display control unit 34, and a data storage unit 35. Each of these units of the notebook PC 17 is a function or means realized by any of the components shown in FIG. 4B operating according to instructions from the CPU 521 according to a program (virtual conference application) expanded from the HD 524 to the RAM 523.

[0216] The virtual conference connection unit 30 connects to the virtual conference service of the virtual conference server 5 and transmits the user name and password, which are authentication data. Also, the virtual conference connection unit 30 manages the URL of the virtual conference space 18 and the conference ID received from the virtual conference service, and connects to the URL of the virtual conference space 18 to perform the process of participating in the virtual conference. Furthermore, the URL of the virtual conference space 18 and the conference ID are transmitted to the VR goggles 10 via the Bluetooth communication control unit 33.

[0217] The remote desktop control unit 31 transmits the screen data of the notebook PC 17 to the virtual conference service in order to display it in the virtual conference space 18.

[0218] The wireless LAN communication control unit 32 executes communication with the virtual conference server 5 via the Wi-Fi router 8.

[0219] The Bluetooth communication control unit 33 executes communication compliant with the Bluetooth standard with the VR goggles 10. The display control unit 34 performs control for displaying the screen data on the display (LCD).

[0220] The data storage unit 35 stores the address (URL) of the virtual conference service provided by the virtual conference server 5, the URL of the virtual conference space 18 received from the virtual conference service, the conference ID, the user name of the user using the notebook PC 17, and the password, etc.

[0221] <<VR Goggles>> The VR goggles 10 have a display control unit 100, a wireless LAN communication control unit 101, a Bluetooth communication control unit 102, a virtual conference connection unit 103, a button information transfer unit 104, and a data storage unit 105. Each of these units of the VR goggles 10 is a function or means realized by operating any of the components shown in FIG. 37 according to an instruction from the CPU 80 according to a program (virtual conference application) expanded from the ROM 82 to the main memory 81.

[0222] The display control unit 100 performs control to display the display data of the virtual whiteboard 18a transmitted from the virtual conference service on the LCD 87. The wireless LAN communication control unit 101 executes communication with the virtual conference server 5 via the Wi-Fi router 8.

[0223] The Bluetooth communication control unit 102 executes communication compliant with the Bluetooth standard with the VR operation controller 11. The virtual conference connection unit 103 performs control to connect to the virtual conference service executed by the virtual conference server 5 using the URL and conference ID of the virtual conference service transmitted from the notebook PC 17.

[0224] The button information transfer unit 104 receives button press information from the VR operation controller 11 and transmits information regarding the pressed button to the virtual conference server 5 via the wireless LAN communication control unit 101.

[0225] The data storage unit 105 stores the address (URL) of the virtual conference service provided by the virtual conference server 5, the conference ID, the user name of the user using the VR goggles 10, the password, and the like.

[0226] <<VR Operation Controller>> The operation controller 11 for VR has a Bluetooth communication control unit 120, a button information transmission unit 121, and an orientation / movement detection unit 122. These functions of the operation controller 11 for VR are functions or means realized by operating according to instructions from the CPU 110 based on a program expanded from the ROM 112 shown in FIG. 38 to the main memory 111.

[0227] The Bluetooth communication control unit 120 executes communication compliant with the Bluetooth standard with the VR goggles 10. The button information transmission unit 121 passes the pressing information of the menu display button 114, the pointer display button 115, and the confirmation button 116 to the Bluetooth communication control unit 120.

[0228] The orientation / movement detection unit 122 detects the orientation (posture in three-dimensional space) and movement of the operation controller 11 for VR from the acceleration and angular velocity data obtained from the six-axis acceleration / angular velocity sensor 113, and passes this information to the Bluetooth communication control unit 120.

[0229] <Meeting reservation> FIG. 41 shows a meeting reservation screen 240 displayed on an arbitrary terminal device. The meeting organizer accesses the meeting reservation server 6 from a web browser of an arbitrary terminal device (such as a PC or a smartphone), reserves a meeting in which the meeting server 4 and the virtual meeting server 5 are coordinated, and registers information regarding the meeting.

[0230] The meeting reservation screen 240 has a meeting name column 241, a host name column 242, a date and time column 243, a participant column 244, a device name column 245, a venue column 246, and a virtual room column 247. The host appropriately enters information in the meeting name column 241, the host name column 242, the date and time column 243, and the participant column 244. The device name column 245 is a column where an electronic blackboard or the like for participating in the meeting is set. Either the venue column 246 or the virtual room column 247 may be set. When the host enters the meeting room name in the venue column 246 and does not enter the room name in the virtual room column 247, a meeting that does not use the virtual meeting server 5 is held. When the host enters the room name in the virtual room column 247 and does not enter the meeting room name in the venue column 246, a meeting that does not use the meeting server 4 is held.

[0231] As shown in FIG. 41, when the host enters and registers both, the participants who participate in Conference Room A and the participants who participate in Virtual Room B can participate in the same meeting. Each user registered as a participant may participate in Conference Room A or Virtual Room B. The virtual room is a three-dimensional space for the meeting.

[0232] Participants can also join the meeting from a device such as the electronic blackboard 7. In this case, the user who uses the electronic blackboard 7 can omit the user authentication by the meeting server 4 using a terminal device such as a PC.

[0233] When the registration button 248 is pressed on the meeting reservation screen 240, the meeting reservation server 6 generates a meeting ID and transmits this meeting ID and the meeting information (the content of the meeting reservation screen 240) to the meeting server 4 and the virtual meeting server 5. When the meeting server 4 and the virtual meeting server 5 receive this meeting information, they store it in the non-volatile memory.

[0234] In addition to this meeting ID and meeting information, the meeting reservation server 6 also sends an email to the participant's email address with the URL for joining the meeting (managed by the meeting server 4) and the URL for the virtual meeting space 18 (managed by the virtual meeting server 5). Note that these pieces of information may be registered in the schedule application used by the participants.

[0235] The meeting server 4 regularly checks (e.g., at 6:00 am every day) the meeting information stored in the non-volatile memory and schedules the meetings scheduled for that day. In the example of FIG. 41, at 14:00 on October 3, 2023, referring to the meeting information shown in FIG. 41, it is confirmed that the virtual room B, which is one of the virtual meeting spaces 18 of the virtual meeting server 5, is registered.

[0236] Then, the virtual meeting connection unit 63 of the meeting server 4 connects to the virtual room B of the virtual meeting server 5 (virtual meeting service) to establish a data transfer session between the meeting server 4 and the virtual room B. Then, the virtual whiteboard synchronization control unit 64 of the meeting server 4 sends the drawing data of the whiteboard area (drawable area 217) on the memory to the virtual room B.

[0237] The virtual meeting service of the virtual meeting server 5 arranges the received drawing data of the whiteboard area at an arbitrary position (an arbitrary position in the three-dimensional space) in the virtual room B as the drawing data of the virtual whiteboard 18a.

[0238] FIG. 42 shows the virtual whiteboard 18a in the virtual room B of the virtual meeting service. Immediately after the start of the meeting, since there is no writing on the whiteboard, only the menu bar 215 is displayed on the virtual whiteboard 18a. The size of the virtual whiteboard 18a may be the size of one page and the user may be able to add pages, or it may be a drawable area 217 with an infinite size.

[0239] <User's Participation in the Meeting> Next, the process when the user participates in the meeting will be described. Users A to C participate in the meeting from Conference Room A. Users A to C use the electronic blackboard 7 to participate in the meeting. When users A to C in Conference Room A touch the "Meeting Participation" button displayed on the electronic blackboard 7, the electronic blackboard 7 connects to the meeting server 4.

[0240] When the meeting server 4 authenticates the electronic blackboard 7, it refers to the meeting information received from the meeting reservation server 6 and determines whether there is a meeting registered that the electronic blackboard 7 can participate in within a certain period from now. If there is a meeting that can be participated in, the meeting server 4 permits the electronic blackboard 7 to participate in this meeting. Permitting participation means that sharing of drawing data with other sites becomes possible.

[0241] User D participates in the meeting from home. User D wears the VR goggles 10 and participates in the meeting (in a visual state as if in Virtual Room B). When User D starts the virtual meeting app on the notebook PC 17, the virtual meeting connection unit 30 connects to the virtual meeting service of the virtual meeting server 5 and displays an input screen for the user name and password. Note that the URL of the virtual meeting service of the virtual meeting server 5 is pre-registered in the notebook PC 17 as the setting data of the virtual meeting app. When User D inputs the user name and password, the virtual meeting connection unit 30 sends these data to the virtual meeting service. The virtual meeting service authenticates this user and sends an authentication OK response to the notebook PC 17. The virtual meeting app on the notebook PC 17 displays a participation button for the virtual meeting. When User D selects the participation button, an input screen for the URL of the virtual meeting space 18 and the meeting ID is displayed. When User D inputs the URL of the virtual meeting space 18 and the meeting ID and presses the OK button, the virtual meeting connection unit 30 sends a connection command including this meeting ID to the URL of the virtual meeting space 18.

[0242] When the virtual meeting service of the virtual meeting server 5 verifies the meeting ID generated previously with the meeting ID and confirms a match, it permits the connection between the notebook PC 17 and the virtual meeting space 18.

[0243] Subsequently, the virtual meeting app on the notebook PC 17 transmits the URL of the virtual meeting space 18 and the meeting ID to the user D's VR goggles 10 via Bluetooth. When the VR goggles 10 receive these data, the virtual meeting connection section 103 transmits a connection command including this meeting ID to the URL of the virtual meeting space 18. When the virtual meeting service of the virtual meeting server 5 verifies the meeting ID with the previously generated meeting ID and confirms a match, it permits the connection between the VR goggles 10 and the virtual meeting space 18.

[0244] When the virtual meeting app on the notebook PC 17 connects to the virtual meeting space 18, it displays a selection button for the remote desktop. When the user D selects this, the remote desktop control section 31 transmits the screen data of the notebook PC 17 to the virtual meeting service of the virtual meeting server 5 to be displayed in the virtual meeting space 18. Note that the virtual display control section 26 of the virtual meeting server 5 controls so that the screen data of the notebook PC 17 displayed in the virtual meeting space 18 is only displayed on the user D's VR goggles 10.

[0245] FIG. 43 is an example of the display of the virtual meeting space 18 displayed on the user D's VR goggles 10. As shown in FIG. 43, a virtual whiteboard 18a and virtual PC screen data 18b are displayed in the virtual meeting space 18.

[0246] User D grasped the VR operation controller 11 with the grip part (the part held by hand) facing upward. The VR operation controller 11 transmits output data (orientation and movement information) from the six-axis acceleration / angular velocity sensor 113 to the virtual conference server 5 via the VR goggles 10. The virtual conference service determines that the VR operation controller 11 is upside down based on the output data. Then, the virtual display control unit 26 of the virtual conference service sets the virtual whiteboard 18a to the handwriting mode and displays the position of the VR operation controller 11 at that time as a pen mark at the upper left corner of the virtual whiteboard 18a. When User D moves the VR operation controller 11 within a vertical plane with respect to the ground in this state, the pen mark also moves in that direction. When User D moves the VR operation controller 11 a predetermined distance in the depth direction (the direction away from User D's body: within a horizontal plane with respect to the ground), the virtual display control unit 26 sets it to the pen-down state. After that, when User D moves the VR operation controller 11 within a vertical plane with respect to the ground and makes a handwriting motion, the virtual display control unit 26 displays the moved trajectory as handwriting data. When User D moves the VR operation controller 11 a predetermined distance in the forward direction (the direction approaching User D's body: within a horizontal plane with respect to the ground), the virtual display control unit 26 sets it to the pen-up state.

[0247] In this way, User D can write characters on the virtual whiteboard 18a using the VR operation controller 11. The synchronization control unit 22 of the virtual conference server 5 transmits this handwriting data to the conference server 4, and the conference server 4 writes this handwriting data into the drawable area 217 of the memory area. Also, the conference server 4 transmits this drawing data to the electronic blackboard 7, and the electronic blackboard 7 displays this drawing data on the display 480.

[0248] Figure 44 shows the virtual whiteboard 18a that displays the drawing data using the VR operation controller 11. The drawing data in Figure 44 consists of two handwritten data 203 of the character "a", a square 205, and a star mark 204.

[0249] When user D presses the pointer display button on the VR operation controller 11, the VR operation controller 11 transmits a pointer display command including the orientation and movement information of its own device to the VR goggles 10 via Bluetooth. The VR goggles 10 transmit the pointer display command to the virtual meeting service of the virtual meeting server 5.

[0250] When the virtual meeting service receives the pointer display command, the pointer position calculation unit 27 calculates the display position of the pointer based on the orientation and movement information of the VR operation controller 11. The pointer position calculation unit 27 displays the pointer 209 on the virtual whiteboard 18a in the virtual meeting space 18. While the user is pressing the pointer display button, the VR operation controller 11 periodically (for example, every 100 milliseconds) transmits a pointer display command including the orientation and movement information of its own device to the VR goggles 10. The VR goggles 10 transmit the pointer display command to the virtual meeting service of the virtual meeting server 5.

[0251] When the virtual meeting service detects that the pointer 209 has moved and there is a change in the display of the virtual whiteboard 18a, it transmits the display data of the virtual whiteboard 18a to the VR goggles 10. While pressing the pointer display button on the VR operation controller 11, user D moves the pointer 209 onto the graphic icon 211 on the menu bar 215 and then presses the confirmation button on the VR operation controller 11. The VR operation controller 11 transmits a confirmation command to the VR goggles 10 via Bluetooth. The VR goggles 10 transmit the confirmation command to the virtual meeting service of the virtual meeting server 5. When receiving this command, the virtual meeting service determines that the graphic icon 211 has been selected and displays the graphic list 212.

[0252] When user D moves the pointer over the square while pressing the pointer display button on the VR operation controller 11 and then presses the OK button, the VR operation controller 11 sends an OK command to the virtual meeting service of the virtual meeting server 5 via the VR goggles 10. When the virtual meeting service receives this OK command, the virtual display control unit 26 determines that the square has been selected and displays the square 205.

[0253] Subsequently, when user D moves the pointer 209 over the star mark while pressing the pointer display button on the VR operation controller 11 and then presses the OK button, the VR operation controller 11 sends an OK command to the virtual meeting service of the virtual meeting server 5 via the VR goggles 10. When the virtual meeting service receives this OK command, the virtual display control unit 26 determines that the star mark has been selected and displays the star mark 204.

[0254] Subsequently, when user D moves the pointer 209 over the shape of the star mark 204 while pressing the pointer display button on the VR operation controller 11 and stops for a predetermined time (e.g., 2 seconds), the shape of the star mark 204 is captured. User D can continue to move the shape of the star mark 204 while pressing the pointer display button.

[0255] Subsequently, user B in conference room A wants to handwrite information related to the drawing data drawn by user D in the empty space on the right. However, since there is little empty space to the right of the handwritten data 203, the square 205, and the star mark 204, user B scrolls the drawable area 217 to the left to expand the empty space and handwrites there.

[0256] The contact position detection unit 70 of the electronic blackboard 7 detects a leftward scroll operation by the user B using the pen, and the display control unit 71 moves the display area. When the user B in the conference room writes by hand or displays a figure using the pen, the contact position detection unit 70 detects the contact position of the pen, and the display control unit 71 generates and displays handwritten data or a figure. The drawing data transmission unit 41 sequentially transmits this drawing data to the conference server 4. The conference server 4 writes the drawing data received from the electronic blackboard 7 into the drawable area 217 on the memory.

[0257] When the user handwrites the handwritten data 208 of "i" and draws the arrow 206 on the electronic blackboard 7, the drawing data of the whiteboard on the memory of the conference server 4 is shown in Fig. 45.

[0258] Fig. 45 shows the drawing data in the drawable area 217 of the conference server 4 including the drawing data drawn by the electronic blackboard 7. Also shown in Fig. 45 is the display area 251 of the electronic blackboard 7. When the conference server 4 receives the drawing data from the electronic blackboard 7 in the conference room, the writing unit 66 writes the drawing data into the drawable area 217 of the whiteboard on the memory.

[0259] The virtual whiteboard synchronization control unit 64 of the conference server 4 transmits the drawing data received from the electronic blackboard 7 to the virtual conference service of the virtual conference server 5. The synchronization control unit 22 of the virtual conference server 5 writes the received drawing data into the virtual whiteboard 18a. Thereby, the whiteboard of the conference server 4 and the virtual whiteboard 18a of the virtual conference server 5 are synchronized. The synchronization control unit 22 of the virtual conference service transmits the screen data (only the difference is sufficient) of the virtual whiteboard 18a to the VR goggles 10 at home.

[0260] Next, user B presses the overall display button 210 on the menu bar 215 on the electronic blackboard 7 to view all the drawing data on the whiteboard. Then, the electronic blackboard 7 sends an overall display command to the conference server 4. When the conference server 4 receives the overall display command, the drawing data grouping processing unit 54 of the conference server 4 groups the drawing data drawn by user B on the electronic blackboard 7 and the drawing data drawn by user D using the VR operation controller 11, respectively.

[0261] FIG. 46 shows group 1 of the drawing data drawn by the VR operation controller 11 and group 2 of the drawing data drawn on the electronic blackboard 7.

[0262] Thereafter, the conference server 4 executes the same processing as the conference server 1 of the first embodiment, and moves the position on the whiteboard (position in the memory) so that the drawing data of group 1 and the drawing data of group 2 enter the display area of the electronic blackboard 7.

[0263] <Operation or process> FIGS. 47 and 48 are sequence diagrams for explaining the processing of drawing the drawing data by the VR operation controller 11 and the processing of drawing the drawing data on the electronic blackboard 7. In the description of FIG. 48, it is assumed that the electronic blackboard 7 and user D have already participated in the conference.

[0264] S101: User D holds the VR operation controller 11 with the grip part (the part held by the hand) facing up, and moves the VR operation controller 11 a predetermined distance in the depth direction (the direction away from user D's body: in the horizontal plane with respect to the ground) (pen down). User D moves the VR operation controller 11 in a vertical plane with respect to the ground to perform a handwriting operation.

[0265] S102: The orientation and motion detection unit 122 of the VR operation controller 11 detects the orientation (posture in three-dimensional space) and motion of the VR operation controller 11 from the acceleration and angular velocity data obtained from the six-axis acceleration and angular velocity sensor 113, and the Bluetooth communication control unit 12 transmits this information to the VR goggles 10.

[0266] S103: The Bluetooth communication control unit 102 of the VR goggles 10 receives the orientation and movement information of the VR operation controller 11, and the wireless LAN communication control unit 101 transmits this information to the virtual conference server 5.

[0267] S104: The communication control unit 23 of the virtual conference server 5 receives this information, and the display data generation unit 25 generates handwritten data from the trajectory of the coordinates in the three-dimensional space based on the orientation and movement of the VR operation controller 11. The virtual display control unit 26 writes the handwritten data to the virtual whiteboard 18a.

[0268] S105: The synchronization control unit 22 of the virtual conference server 5 transmits this handwritten data to the conference server 4. S105-2: The communication control unit 23 of the virtual conference server 5 transmits this handwritten data to the VR goggles 10. S105-3: The wireless LAN communication control unit 101 of the VR goggles 10 receives the handwritten data, and the display control unit 100 displays the handwritten data.

[0269] S106: The drawing data receiving unit 53 of the conference server 4 receives the handwritten data, and the writing unit 66 writes it to the drawable area 217 of the whiteboard.

[0270] S107: The drawing data transmitting unit 57 of the conference server 4 transmits this handwritten data to the electronic blackboard 7.

[0271] S108: The drawing data receiving unit 42 of the electronic blackboard 7 receives the handwritten data, and the display control unit 71 displays the handwritten data on the display 480.

[0272] S109~S116: While the user D is handwriting with the VR operation controller 11, the same processing is repeatedly executed.

[0273] S117: Next, user B (who may be A or C) in conference room A scrolled the drawable area 217 to the left to expand the free space.

[0274] S118: The contact position detection unit 70 of the electronic blackboard 7 detects the leftward scroll operation by user B using the pen, and the display control unit 71 moves the display area.

[0275] S119: User B in the conference room writes by hand or displays a figure using the pen.

[0276] S120: The contact position detection unit 70 detects the contact position of the pen, and the display control unit 71 generates and displays handwritten data or a figure.

[0277] S121: The drawing data transmission unit 41 of the electronic blackboard 7 sequentially transmits this drawing data to the conference server 4.

[0278] S122: The drawing data reception unit 53 of the conference server 4 receives the drawing data, and the writing unit 66 writes the drawing data received from the electronic blackboard 7 into the drawable area 217 of the whiteboard on the memory.

[0279] S123: The virtual whiteboard synchronization control unit 64 of the conference server 4 transmits the drawing data received from the electronic blackboard 7 to the virtual conference service of the virtual conference server 5.

[0280] S124: The communication control unit 23 of the virtual conference service receives the drawing data, and the synchronization control unit 22 writes the drawing data into the virtual whiteboard 18a.

[0281] S125: The synchronization control unit 22 of the virtual conference service transmits the display data (only the difference is sufficient) of the virtual whiteboard 18a to the home VR goggles 10 via the communication control unit 23.

[0282] S126: The wireless LAN communication control unit 101 of the VR goggles 10 receives the display data of the virtual whiteboard 18a, and the display control unit 100 displays the display data of the virtual whiteboard 18a.

[0283] S127~S134: While user B (it may also be A or C) writes on the electronic blackboard 7, the same processing is repeatedly executed.

[0284] S135: Next, user B (it may also be A or C) presses the overall display button 210 on the menu bar 215 displayed on the electronic blackboard 7.

[0285] S136: The contact position detection unit 70 of the electronic blackboard 7 detects the pressing of the overall display button 210 based on the contact position. The drawing data transmission unit 41 transmits an overall display command to the conference server 4.

[0286] S137: When the drawing data reception unit 53 of the conference server 4 receives the overall display command, the drawing data grouping processing unit 54 of the conference server 4 groups the drawing data drawn by user B on the electronic blackboard 7 and the drawing data drawn by user D using the VR operation controller 11, respectively.

[0287] S138: The drawing data movement unit 56 moves the drawable area 217 so that there is no gap between the drawing data of group 1 and the drawing data of group 2 (so that it enters the display area of the electronic blackboard 7), similar to the first embodiment.

[0288] S139: The drawing data transmission unit 57 of the conference server 4 transmits the drawing data with the group moved in the drawable area 217 to the electronic blackboard 7.

[0289] S140: The drawing data reception unit 42 of the electronic blackboard 7 receives the drawing data, and the display control unit 71 causes the display to display the entire drawing data.

[0290] Note that step S136 corresponds to the processing of the first embodiment, but the conference server 4 may perform the processing described in the second to fourth embodiments.

[0291] <Main effects> According to the present embodiment, even when a virtual meeting is held, drawing data drawn at a plurality of bases can be displayed in a list.

[0292] Note that, also in the present embodiment, the display area may be switched in page units as in the sixth embodiment described below. The virtual whiteboard 18a secures the number of pages. The drawing data of each page written on each page is displayed in a list by the process of the sixth embodiment.

[0293] [Sixth Embodiment] In the first to fifth embodiments, the electronic blackboard 2 capable of moving the display area by scrolling the screen of the device has been described. In the present embodiment, an electronic blackboard 2 that cannot move the display area is assumed. The electronic blackboard 2 treats the screen size (display area) as one page. Then, regarding the drawing data drawn on a plurality of pages, a conference server that arranges and displays all the drawing data of all the pages so as to fit in the display area of the electronic blackboard 2 will be described.

[0294] <System configuration example> FIG. 49 shows a system configuration diagram of the remote conference system 600 in the present embodiment. In the description of FIG. 49, mainly the differences from FIG. 2 will be described. In the remote conference system 600 of FIG. 49, both bases 1 and 2 are conference rooms, an electronic blackboard 2A is arranged at base 1, and an electronic blackboard 2B instead of a tablet PC is arranged at base 2. The electronic blackboard 2A and the electronic blackboard 2B are connected to the conference server 301.

[0295] <Regarding functions> FIG. 50 shows a functional block diagram of the conference server 301 in the present embodiment. Since the functional block diagram of the electronic blackboard 2 is the same as that in the first embodiment, only the functional block diagram of the conference server 301 is shown. In the description of FIG. 50, the differences from FIG. 13 will be mainly described. The conference server 301 in FIG. 50 newly has a page management unit 65. The page management unit 65 is a means for creating a new page in response to a request from the electronic blackboard 2. Specifically, the page management unit 65 manages drawing data such as handwritten data in page units.

[0296] The drawing data group processing unit 54 is a means for grouping drawing data for each page to create a group. The handwritten character detection unit 60 is a means for detecting characters from the drawing data for each page. The drawing data enlargement / reduction unit 61 calculates a first average of the character sizes for each group, calculates a second average of the character sizes between the groups based on the first average, and enlarges or reduces the drawing data of each page by the ratio of the second average to the first average. The drawing data movement unit 56 arranges the enlarged or reduced drawing data of each group so as not to overlap in the area corresponding to the display area of a predetermined device.

[0297] <Participation in the Conference> By the operations of users A to C at Site 1, the electronic blackboard 2A connects to a conference registered in advance in the conference server 301. The electronic blackboard 2A transmits the device information of the electronic blackboard 2A to the conference server 301, and the conference server 301 permits the electronic blackboard 2A to participate in the conference. Also at Site 2, by the operation of the user, the electronic blackboard 2B connects to a conference registered in advance in the conference server 301. At this time, the device information of the electronic blackboard 2B is transmitted to the conference server 301, and the conference server 301 permits the electronic blackboard 2B to participate in the conference.

[0298] Subsequently, users A to C at Site 1 start the whiteboard application on the electronic blackboard 2A. The electronic blackboard 2A displays a display area where drawing data is drawn on the display 480, and transmits a whiteboard start command including the number of display pixels (the number of pixels in the vertical direction and the number of pixels in the horizontal direction) of the display area to the conference server 301, entering the whiteboard sharing mode with the electronic blackboard 2B at other sites.

[0299] When the conference server 301 receives the whiteboard start command, it stores in the memory the number of display pixels included in the whiteboard start command in association with the device information of the electronic blackboard 2A, and secures a memory area for the whiteboard with the size of this number of display pixels. The conference server 301 also manages the whiteboard area on the memory as the first page.

[0300] When users D and E at Site 2 also start the whiteboard application on the electronic blackboard 2B, the electronic blackboard 2B displays the whiteboard display area on the display, and transmits a whiteboard start command including the number of display pixels of the display area to the conference server 301, entering the whiteboard sharing mode with the electronic blackboard 2A at other sites. When the conference server 301 receives the whiteboard start command, it stores in the memory the number of display pixels included in the whiteboard start command in association with the device information of the electronic blackboard 2B. The conference server 301 compares the number of display pixels of the electronic blackboards 2A and 2B and determines that they are the same (when the electronic blackboards 2A and 2B are of the same model).

[0301] <Sharing of Drawing Data> On the electronic blackboard 2A at Site 1, for example, when user A writes by hand, the contact position detection unit 45 detects the contact position, and the display control unit 44 generates and displays handwritten data at that position. User A also selects a square from the graphic list 212 displayed by pressing the graphic icon 211 in the menu bar 215. The electronic blackboard 2A generates and displays this square. The drawing data transmission unit 41 sequentially transmits this drawing data to the conference server 301.

[0302] When the conference server 301 receives drawing data from Site 1, it writes this drawing data to the page of the currently shared whiteboard in the memory and also transmits this drawing data to the electronic blackboard 2B at Site 2.

[0303] FIG. 51 shows the drawing data drawn on each page of the whiteboard. FIG. 51(a) shows the drawing data of page 1, FIG. 51(b) shows the drawing data of page 2, FIG. 51(c) shows the drawing data of page 3, and FIG. 51(d) shows the drawing data of page 4. The drawing data of page 1 includes two handwritten data 203 of "あ" handwritten by the user at Site 1 and a square 205 drawn by the user at Site 2.

[0304] Next, user D at Site 2 wants to handwrite information related to the drawing data at Site 1 in the empty space on the right side of the handwritten data 203 of "あ". However, since there is little empty space, user D touches the page forward button 254 on the menu bar 215.

[0305] FIG. 52 shows an example of the menu bar 215. The menu bar 215 has, in addition to the overall display button 210, a page back button 255 and a page forward button 254. The page forward button 254 accepts an operation to switch to the next page, and the page back button 255 accepts an operation to switch to the previous page. When the page forward button 254 is pressed while the last page is being displayed, a page with nothing drawn is added.

[0306] When the page forward button 254 is pressed on the electronic blackboard 2B, it transmits a next page switching command to the conference server 301. When the conference server 301 receives the next page switching command, the page management unit 65 secures the whiteboard area of the second page in the memory, initializes the whiteboard area, and transmits a new display area (page) creation command to the electronic blackboards 2A and 2B. The electronic blackboards 2A and 2B each display a new display area (page) on the display 480. Therefore, the area other than the menu bar 215 becomes blank.

[0307] When user D writes on the electronic blackboard 2B by hand, the contact position detection unit 45 detects the contact position, and the display control unit 44 generates and displays handwritten data at that position. The drawing data transmission unit 41 sequentially transmits this handwritten data to the conference server 301.

[0308] When the conference server 301 receives the handwritten data from site 2, it writes this handwritten data into the whiteboard area on the second page in the memory, and at the same time, transmits this handwritten data to the electronic blackboard 2A at site 1. The electronic blackboard 2A at site 1 receives and displays the handwritten data. Fig. 51(b) shows the drawing data on the second page. In Fig. 51(b), 18 handwritten data 225 of "i" are drawn.

[0309] Since user B at site 1 wants to write further by hand, when pressing the page forward button 254 on the menu bar 215, the electronic blackboard 2A transmits a next page switching command to the conference server 301. When the conference server 301 receives the next page switching command, the page management unit 65 secures the whiteboard area on the third page in the memory, initializes the whiteboard area, and transmits a new display area (page) creation command to the electronic blackboards 2A and 2B. The electronic blackboards 2A and 2B respectively display the new display area (page) on the display 480. Therefore, the area other than the menu bar 215 becomes blank.

[0310] When user B writes on the electronic blackboard 2A by hand, the contact position detection unit 45 detects the contact position, and the display control unit 44 generates and displays handwritten data at that position. The drawing data transmission unit 41 sequentially transmits this handwritten data to the conference server 301. When the conference server 301 receives the handwritten data from site 1, it writes this handwritten data into the whiteboard area on the third page in the memory, and at the same time, transmits this handwritten data to the electronic blackboard 2B at site 2. The electronic blackboard 2B at site 2 receives and displays the handwritten data. Fig. 51(c) shows the drawing data on the third page. In Fig. 51(c), 3 handwritten data 231 of "u" are drawn.

[0311] After that, since user E at base 2 wants to write by hand further, when the user presses the page forward button 254 of the whiteboard on the menu bar 215, the electronic blackboard 2B sends a next page switching command to the conference server 301. When the conference server 301 receives the next page switching command, the page management unit 65 secures the whiteboard area of the fourth page in the memory, initializes the data in the whiteboard area, and sends a new creation command for the display area (page) to the electronic blackboards 2A and 2B. The electronic blackboards 2A and 2B respectively display the new display area (page) on the display 480. Therefore, the area other than the menu bar 215 becomes blank paper.

[0312] When user E writes by hand on the electronic blackboard 2B, the contact position detection unit 45 detects the contact position, and the display control unit 44 generates and displays handwritten data at that position. User E also selects a star mark from the graphic list 212 displayed by touching the graphic icon 211 on the menu bar 215. The electronic blackboard 2B generates and displays the star mark graphic. The drawing data transmission unit 41 sequentially sends the handwritten data and the drawing data of the graphic to the conference server 301. When the conference server 301 receives the drawing data from base 2, it writes this drawing data into the whiteboard area of the fourth page in the memory and also sends this drawing data to the electronic blackboard 2A at base 1. The electronic blackboard 2A at base 1 receives and displays the drawing data. FIG. 51(d) shows the drawing data of the fourth page. Four handwritten data 232 of "え" and a star mark 233 are drawn in FIG. 51(d).

[0313] Currently, the drawing data for the fourth page is being displayed on the display 480 of the electronic blackboard 2A. The drawing data for the fourth page is also being displayed on the display 480 of the electronic blackboard 2B. Here, since we also want to display and check the drawing data from the first page to the third page, the user at Site 1 pressed the overall display button 210. When the electronic blackboard 2A detects the pressing of the overall display button 210, it sends an overall display command to the conference server 301. When the conference server 301 receives the overall display command, the drawing data grouping processing unit 54 of the conference server 301 groups the drawing data for each page respectively.

[0314] Figure 53 shows the grouped drawing data for each page. The drawing data grouping processing unit 54 creates an outer bounding rectangle for the drawing data for each page (not for each site), and sets the drawing data enclosed by the rectangular area as one group. Figure 53(a) shows Group 1 of the drawing data for Page 1, Figure 53(b) shows Group 2 of the drawing data for Page 2, Figure 53(c) shows Group 3 of the drawing data for Page 3, and Figure 53(d) shows Group 4 of the drawing data for Page 4.

[0315] Next, as shown in Figure 54, the handwritten character detection unit 60 of the conference server 301 extracts the characters within each group. Figure 54 is a diagram explaining the characters extracted from within each group. 'あ' (a), 'い' (i), 'う' (u), and 'え' (e) have been detected from Groups 1 to 4 respectively. The handwritten character detection unit 60 calculates the average size of the characters for each group. The size of the character is represented by the number of points corresponding to the size of the extracted character.

[0316] The drawing data enlargement / reduction unit 61 calculates the average of the sizes of the characters extracted from the drawing data for Pages 1 to 4 for each group. · The drawing data enlargement / reduction unit 61 extracts the two characters on Page 1 and obtains the average of the number of points corresponding to the size of each character. Let the average of the characters on Page 1 be a1. · The drawing data enlargement / reduction unit 61 extracts the 18 characters on Page 2 and obtains the average of the number of points corresponding to the size of each character. Let the average of the characters on Page 2 be a2. · The drawing data enlargement / reduction unit 61 extracts the three characters on page 3 and obtains the average of the number of points corresponding to the size of each character. Let the average of the characters on page 3 be a3. · The drawing data enlargement / reduction unit 61 extracts the four characters on page 4 and obtains the average of the number of points corresponding to the size of each character. Let the average of the characters on page 4 be a4.

[0317] The average W of the averages a1 to a4 of the sizes of the characters on pages 1 to 4 is W = (a1 + a2 + a3 + a4) / 4.

[0318] The drawing data enlargement / reduction unit 61 of the conference server 301 enlarges or reduces the drawing data of each site as follows using the average W. · Since the average (a1) of the sizes of the characters on page 1 is larger than W, the drawing data (including graphics) on page 1 is reduced by W / a1 (reduced for each group). · Since the average (a2) of the sizes of the characters on page 2 is smaller than W, the drawing data on page 2 is enlarged by W / a2 (enlarged for each group). · Since the average (a3) of the sizes of the characters on page 3 is larger than W, the drawing data on page 3 is reduced by W / a3 (reduced for each group). · Since the average (a4) of the sizes of the characters on page 4 is larger than W, the drawing data (including graphics) on page 4 is reduced by W / a4 (reduced for each group).

[0319] Figure 55 shows the drawing data after enlargement or reduction. It can be seen that the drawing data of pages 1, 3, and 4 has become smaller, and the drawing data of page 2 has become larger.

[0320] Next, the drawing data moving unit 56 of the conference server 301 arranges the enlarged or reduced drawing data of pages 1 to 4 close to each other and side by side. Separately from pages 1 to 4, the drawing data moving unit 56 secures a memory area (hereinafter referred to as the integration area 260) large enough to write the drawing data of pages 1 to 4, and arranges (copies) the drawing data of pages 1 to 4 so as not to overlap. The integration area 260 may be the maximum size capable of writing the drawing data of pages 1 to 4, and the size may be adjusted by the circumscribed rectangle of the drawing data of pages 1 to 4 after writing. Note that a machine learning model may be used for the process of arranging a plurality of groups so as not to overlap.

[0321] FIG. 56 shows the drawing data of pages 1 to 4 written in the integration area 260. The drawing data moving unit 56 moves group 2, for example, so that the upper right corner of group 1 and the upper left corner of group 2 substantially coincide. The drawing data moving unit 56 moves group 3, for example, so that the lower right corner of group 1 and the upper right corner of group 3 substantially coincide (simply making the upper side of group 3 substantially coincide with the bottom side of group 1 may also be acceptable). The drawing data moving unit 56 moves group 4, for example, so that the bottom side of group 2 and the upper side of group 4 substantially coincide.

[0322] Next, as shown in FIG. 57, the drawing data moving unit 56 of the conference server 301 divides the integration area 260 into four parts. FIG. 57 shows the integration area 260 divided into four parts. Areas 1 to 4 are obtained by the division. The drawing data moving unit 56 moves the drawing data of each group so that it fits within the range of each of areas 1 to 4. That is, the drawing data moving unit 56 divides into the same number of areas as the number of pages. However, when the number of pages is odd, it may be made an even number that is easier to divide by adding 1. In FIG. 57, pages 1 to 4 correspond to areas 1 to 4.

[0323] After the division, the drawing data moving unit 56 determines whether the drawing data of the page fits within the range of the area.

[0324] The drawing data moving unit 56 determines whether Group 1 fits within the range of Region 1. In FIG. 57, it is determined that Group 1 is within the range of Region 1.

[0325] Next, the drawing data moving unit 56 determines whether Group 2 fits within the range of Region 2. In FIG. 57, it is determined that Group 2 is larger than Region 2 in both width and height.

[0326] Next, the drawing data moving unit 56 determines whether Group 3 fits within the range of Region 3. In FIG. 57, it is determined that Group 3 is smaller than the four - divided region in both width and height. Similarly, it is determined that Group 4 is smaller than Region 4 in both width and height.

[0327] Subsequently, for Group 3 and Group 4, which are determined to be smaller than the four - divided region in both width and height, the drawing data moving unit 56 determines whether they overlap with the dividing line 261 of the four - division. Since the size of Group 1 is smaller than Region 1, there is no possibility of overlapping with the dividing line 261. In FIG. 57, it is determined that Group 3 overlaps with the horizontal dividing line 261. The drawing data moving unit 56 determines whether there is drawing data (of the group) below Group 3 within the integrated region 260. If the drawing data moving unit 56 determines that there is no drawing data below Group 3, it moves Group 3 downward until the upper side of Group 3 overlaps with the horizontal dividing line 261.

[0328] FIG. 58 shows the integrated region 260 in a state where Group 3 has been moved downward. In FIG. 58, the upper side of Group 3 overlaps with the horizontal dividing line 261.

[0329] Rather than moving the drawing data of Groups 1 - 4 after dividing into Regions 1 - 4, by moving the drawing data of Groups 1 - 4 first and then moving them to Regions 1 - 4, the drawing data of each page can be arranged without gaps, and it becomes easier to arrange one page in one region.

[0330] Subsequently, the drawing data moving unit 56 determines whether or not the integrated area 260 is within the display area of the electronic blackboard 2A. If it is within the range of the display area, the drawing data transmitting unit 57 transmits the drawing data within the integrated area 260 shown in FIG. 58 to the electronic blackboards 2A and 2B. The electronic blackboards 2A and 2B display the received drawing data. At this time, if the integrated area 260 is larger than the display area of the electronic blackboard 2 that requested the list display of the drawing data, the drawing data enlargement / reduction unit 61 reduces the drawing data of each group arranged in the integrated area 260 so as to fit into the display area of the electronic blackboard 2 that requested the list display of the drawing data. The drawing data transmitting unit 57 transmits the reduced drawing data to the electronic blackboards 2A and 2B. The electronic blackboards 2A and 2B display the received drawing data of the integrated area 260. Incidentally, if the integrated area 260 is larger than the display area of the electronic blackboard 2 that requested the list display of the drawing data, the drawing data enlargement / reduction unit 61 may regard the data in the area of the integrated area 260 as one image data and reduce it by image reduction processing (such as thinning out pixel data) so as to fit into the display area of the electronic blackboard 2 that requested the list display of the drawing data.

[0331] <Operation or process> FIGS. 59 to 61 are sequence diagrams for explaining the process in which the conference server 301 causes the electronic blackboards 2A and 2B to display a list of drawing data of a plurality of pages. In the description of FIGS. 59 to 61, the differences from FIG. 12 will mainly be explained. First, the processes of steps S201 to S212 may be the same as those of steps S1 to S12 in FIG. 12.

[0332] S213: Next, since user D at site 2 wants to write by hand further, the user presses the page forward button 254 on the menu bar 215.

[0333] S214: The contact position detection unit 45 of the electronic blackboard 2B detects the pressing of the page forward button 254, and the LAN communication control unit 96 transmits the next page switching command to the conference server 301.

[0334] S215: When the conference control unit 55 of the conference server 301 receives the next page switching command, the page management unit 65 secures a whiteboard area for one page in the memory.

[0335] S216: The conference control unit 55 of the conference server 301 initializes the whiteboard area and sends a new page creation command for the display area (page) to the electronic blackboards 2A and 2B.

[0336] S217, S218: The LAN communication control units 96 of the electronic blackboards 2A and 2B respectively receive the new page creation command for the display area (page), and the display control unit 44 displays the new page on the display 480.

[0337] Subsequently, the processing of steps S219 to S230 may be the same as that of steps S15 to S26 in FIG. 12.

[0338] Move to FIG. 60 for explanation. S241 to S246: When user B at site 1 adds a page, the same processing as steps S213 to S218 is performed in steps S241 to S246.

[0339] Subsequently, the processing of steps S247 to S258 may be the same as that of steps S201 to S212.

[0340] S259 to S264: When user D at site 2 adds a page, the same processing as steps S213 to S218 is performed in steps S259 to S264.

[0341] Subsequently, the processing of steps S265 to S276 may be the same as that of steps S219 to S230.

[0342] Move to FIG. 61 for explanation. S281: Currently, the fourth page is being displayed on the displays 480 of the electronic blackboards 2A and 2B. Since user B at site 1 wants to also display and check the drawing data from the first page to the third page, the overall display button 210 was pressed.

[0343] S282: The contact position detection unit 45 of the electronic blackboard 2A detects the pressing of the overall display button 210, and the drawing data transmission unit 41 transmits the overall display command to the conference server 301.

[0344] S283: When the drawing data reception unit 53 of the conference server 301 receives the overall display command, the drawing data group processing unit 54 groups the drawing data of each page respectively.

[0345] S284: Next, the handwritten character detection unit 60 detects characters from each group and calculates the average of the character sizes within the group.

[0346] S285: The drawing data enlargement / reduction unit 61 compares the average of the character sizes of all groups with the average of the character sizes of each group to calculate the enlargement or reduction rate, and enlarges or reduces the character sizes of each group.

[0347] S286: The drawing data movement unit 56 secures the integration area 260 and arranges the enlarged or reduced groups within the integration area 260 so that they do not overlap.

[0348] S287: The drawing data enlargement / reduction unit 61 compares the size of the display area of the electronic blackboard 2A with the size of the integration area 260. If the size of the integration area 260 is less than or equal to the display area of the electronic blackboard 2A, proceed to step S288. If the size of the integration area 260 is larger than the display area of the electronic blackboard 2A, the drawing data enlargement / reduction unit 61 reduces the integration area 260 so that it fits within the display area of the electronic blackboard 2A (while maintaining the aspect ratio so that both the vertical and horizontal dimensions fit).

[0349] S288: The drawing data transmission unit 57 transmits the drawing data of the integration area 260 to the electronic blackboard 2A.

[0350] S289: The drawing data reception unit 42 of the electronic blackboard 2A receives the drawing data, and the display control unit 44 causes the received drawing data to be displayed on the display 480.

[0351] <Main effects> According to this embodiment, even when devices such as the electronic blackboards 2A and 2B switch the display area in page units, each of the electronic blackboards 2A and 2B can arbitrarily display a list of the drawing data for each page.

[0352] [Other application examples] As described above, the best mode for carrying out the present invention has been described using examples. However, the present invention is not limited to such examples, and various modifications and substitutions can be made without departing from the gist of the present invention.

[0353] For example, in this embodiment, the user writes by hand on an electronic blackboard or the like, but voice operation may be used.

[0354] In each embodiment, the user who presses the overall display button 210 may be able to select, for example, from the thumbnails of each site the site to be displayed in a list. Further, the user who presses the overall display button 210 may be able to select, for example, from the thumbnails of each page the page to be displayed in a list.

[0355] Also, the user who presses the overall display button 210 may be able to specify a reduction ratio or an enlargement ratio for each site or page to be displayed in a list by the user.

[0356] Also, in this embodiment, a list of the drawing data is transmitted only to the electronic blackboard 2 on which the overall display button 210 has been pressed, but a list of the drawing data may be displayed on the devices at each site. A device on which the overall display button 210 has not been pressed displays a notification that a list of the drawing data has been displayed, and when the user performs an operation to display it, the list of the drawing data is displayed.

[0357] In addition, the electronic blackboard may be referred to as an electronic whiteboard, an electronic information board, or the like. Further, the present embodiment can be preferably applied not only to an electronic blackboard but also to an information processing apparatus having a touch panel. Examples of the information processing apparatus equipped with a touch panel include a PC having a touch panel, a tablet terminal, and a smartphone. These are generally general-purpose information processing apparatuses, but when an application that functions as a display device is executed, the user can operate them as a display device.

[0358] In addition, the configuration examples such as those in FIG. 5 are divided according to the main functions in order to facilitate the understanding of the processing by the conference servers 1, 4, and 301. The present invention is not limited by the way of dividing the processing units or the names. The processing of the conference servers 1, 4, and 301 can be further divided into more processing units according to the processing content. Also, one processing unit can be divided so as to include more processing.

[0359] In addition, each function of the above-described embodiment can be realized by one or a plurality of processing circuits. Here, the "processing circuit" in this specification refers to a processor programmed to execute each function by software like a processor implemented by an electronic circuit, an ASIC (Application Specific Integrated Circuit) designed to execute each function described above, a DSP (digital signal processor), an FPGA (field programmable gate array), or a device such as a conventional circuit module.

[0360] Embodiments of the present invention bring about significant improvements in the capabilities and functionality of computers. These improvements enable users to utilize a computer that provides a more efficient and robust interaction with a table, which is a method of storing and presenting information in an information processing device. Furthermore, embodiments of the present invention provide a better user experience through the use of a more efficient, powerful, and robust user interface. Such a user interface provides better interaction between humans and machines.

Explanation of Signs

[0361] 1, 4, 301 Conference Server 2, 2A, 2B, 7 Electronic Blackboard 3 Tablet PC 5 Virtual Conference Server 6 Conference Reservation Server 10 VR Goggles 11 VR Operation Controller 17 Notebook PC 300, 600 Remote Conference System 700 Virtual Conference System

Prior Art Documents

Patent Documents

[0362]

Patent Document 1

Claims

1. An information processing apparatus capable of communicating with a plurality of devices that display any display area of a drawable area or switch the display area in page units via a network, a drawing data receiving unit that receives drawing data drawn in the display area from the plurality of devices, a writing unit that writes the received plurality of drawing data into one of the drawable areas or the same page as the device, a drawing data moving unit that moves at least a part of the drawing data of the drawable area to an area corresponding to the display area of a predetermined device and arranges the drawing data of the plurality of devices in the area, or arranges the drawing data of a plurality of pages in the area, a drawing data transmitting unit that transmits the drawing data of each device arranged in the area by the drawing data moving unit to the predetermined device, An information processing apparatus having the above.

2. having a drawing data grouping processing unit that groups the drawing data written in the drawable area for each device, The information processing apparatus according to claim 1, wherein when there is a free space between the first group and the second group, the drawing data moving unit moves the first group or the second group so that the free space disappears.

3. a handwritten character detection unit that detects characters from the grouped drawing data for each device, obtains a ratio of an average of the sizes of the characters detected from the first group and an average of the sizes of the characters detected from the second group, and when the ratio exceeds a threshold value, adjusts the sizes of the drawing data of the first group and the second group so that the ratio becomes the threshold value. A drawing data enlarging and reducing unit, The information processing apparatus according to claim 2, having the above.

4. the display area of the first device that requested the list display of the drawing data is landscape, in the drawable area, below the drawing data drawn in the display area of the first device, the drawing data is written by a second device, and when the drawing data of the first device and the second device does not fit in the display area of the first device, The information processing apparatus according to claim 2, wherein the drawing data moving unit moves a second group of the drawing data drawn by the second device beside a first group of the drawing data drawn by the first device.

5. If the drawing data of the entire first group and the moved second group does not fit into the display area of the first device, it has a drawing data enlargement / reduction unit that reduces the size of the entire drawing data so that the entire drawing data fits into the display area of the device that requested the list display of the drawing data. The information processing apparatus according to claim 4. **Claim 6** The drawing data receiving unit has received the drawing data from N devices, a drawing data grouping processing unit that groups the drawing data from the N devices written in the drawable area for each device, a handwritten character detection unit that detects characters from the drawing data of each group, calculates a first average of the sizes of the characters for each group, calculates a second average of the sizes of the characters between the groups based on the first average, and has a drawing data enlargement / reduction unit that enlarges or reduces the drawing data of each group by the ratio of the second average to the first average. The drawing data moving unit evenly divides the display area of the device that requested the list display of the drawing data into N areas, and arranges the enlarged or reduced drawing data of each group in each area. The information processing apparatus according to claim 1. **Claim 7** The drawing data moving unit arranges the drawing data of each group, enlarged or reduced so that the upper left corners of each group coincide, at the upper left corners of the N areas. When the first drawing data protrudes from the area assigned to an arbitrary group into another area, the drawing data moving unit moves the second drawing data in the other area by the amount that the first drawing data protrudes in the direction in which the first drawing data protrudes in the other area. The information processing apparatus according to claim 6. **Claim 8** When the group moved to the right area does not fit horizontally into the display area of the device that requested the list display of the drawing data, and there is a vacant space between the group moved to the left area and the group moved to the right area having the same height as the left area, the drawing data moving unit moves the right group leftward so that the vacant space disappears. The information processing apparatus according to claim 7. **Claim 9** a page management unit that creates a new page in response to a request from the device A drawing data grouping processing unit that groups the drawing data for each page to create groups; A handwritten character detection unit that detects characters from the drawing data for each page; Calculating a first average of the sizes of the characters for each group; Calculating a second average of the sizes of the characters between the groups based on the first average; A drawing data enlargement / reduction unit that enlarges or reduces the drawing data of each page by the ratio of the second average to the first average; The information processing apparatus according to claim 1, wherein the drawing data moving unit arranges the drawing data of each enlarged or reduced group in the secured integrated area so as not to overlap.

10. When the integrated area is larger than the display area of the device that requested the list display of the drawing data, The information processing apparatus according to claim 9, wherein the drawing data enlargement / reduction unit reduces the drawing data of each group arranged in the integrated area so as to fit within the display area of the device that requested the list display of the drawing data.

11. Virtual It can communicate with a virtual conferencing service that causes the device to display the drawing data displayed in the virtual space, When the drawing data transmitted from the virtual conferencing service is written in the drawable area, or when the drawing data transmitted from the virtual conferencing service is drawn on any of a plurality of pages, When a list display of the drawing data is requested from a predetermined device, The drawing data moving unit moves at least a part of the drawing data in the drawable area to an area corresponding to the display area of the predetermined device and arranges the drawing data of the plurality of devices in the area, or arranges the drawing data of a plurality of pages in the area, The information processing apparatus according to any one of claims 1 to 10, wherein the drawing data transmitting unit transmits the drawing data of each device arranged in the area by the drawing data moving unit to the predetermined device.

12. An information display method performed by an information processing apparatus capable of communicating with a plurality of devices that display an arbitrary display area of a drawable area or switch the display area in page units via a network, A process of receiving drawing data drawn in the display area from a plurality of the devices; A process of writing the received plurality of drawing data in one of the drawable areas or on the same page as the device; Moving at least a part of the drawing data of the drawable area to an area corresponding to the display area of a predetermined device and arranging the drawing data of the plurality of devices in the area, or arranging the drawing data of a plurality of pages in the area; Transmitting the drawing data of each device arranged in the area to the predetermined device; An information display method for performing the above.

Images