Equipment systems, devices, terminal equipment, image generation methods, programs

Abstract

To provide a technique capable of sharing the state of an operator operating a display device with a terminal device in another foothold.SOLUTION: An apparatus system includes: an input display device for displaying an input display area; an imaging apparatus for imaging an operator image including the input display device and a person inputting to the input display device; and an information processing system including an acquisition part for acquiring the operator image and a sharing part for allowing the input display device and a terminal device to share the input display area. The information processing system includes a transmission part for transmitting the operator image to the terminal device, to display it on the input display area.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present invention relates to a device system, a device, a terminal device, an image generation method, and a program.

Background Art

[0002] An input display device that displays data handwritten on a touch panel by an input means such as a pen or a finger is known. An input display device equipped with a relatively large touch panel is arranged indoors or outdoors and is used as an electronic blackboard or the like by a plurality of users.

[0003] A technique for estimating an operator of an input display device using a camera mounted on an electronic blackboard is known (for example, see Patent Document 1). Patent Document 1 discloses a technique for estimating a user who is included in a first image but not included in a second image as a current touch panel operator.

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, the conventional technology has a problem that, in response to a predetermined action of a user, an image of a partial region cannot be generated from a surrounding image based on a range indicated by a plurality of the specific images.

[0005] In view of the above problems, an object of the present invention is to provide a device system that can generate an image of a partial region from a surrounding image based on a range indicated by a plurality of the specific images in response to a predetermined action of a user.

Means for Solving the Problems

[0006] In view of the above problems, the present invention provides Remote conference a photographing unit that photographs an image, Participants of the aforementioned remote meeting a detection unit that detects an action, and in response to the action of the participant, based on the Remote conference image included in the Determine the range image, based on the Remote conference image, from the The object and the participantThe system comprises an image generation unit that generates an image of a portion of a region, and a display unit that displays the image of the portion of the region, wherein the image generation unit On the aforementioned object Multiple of the above Determine the range An image of a portion of the region is generated based on the range indicated by the image. Characterized by . [Effects of the Invention]

[0007] This technology can provide a method for generating an image of a portion of a region from a surrounding image based on the range indicated by a plurality of specific images, in response to a predetermined action by the user. [Brief explanation of the drawing]

[0008] [Figure 1] This diagram outlines the creation of recording information that saves the screen of an application running during a remote meeting, along with a panoramic image of the surroundings. [Figure 2] This diagram illustrates the general process by which objects displayed on an electronic whiteboard are displayed on terminal devices at other locations, along with the operator's image. [Figure 3] This diagram shows an example configuration of a record information creation system. [Figure 4] This figure shows an example of the hardware configuration of an information processing system and terminal device. [Figure 5] This figure shows an example of a hardware configuration for a meeting device. [Figure 6] This diagram illustrates the imaging range of the meeting device. [Figure 7] This diagram illustrates the process of extracting images from a panoramic image and a speaker image. [Figure 8] This is a hardware configuration diagram of an example of an interactive whiteboard. [Figure 9] This is an example of a functional block diagram that explains the functions of terminal devices, meeting devices, and information processing systems in a record information creation system by dividing them into blocks. [Figure 10] This figure shows an example of video recording information stored in the information storage unit. [Figure 11]This is a diagram showing an example of meeting information managed by the communication management department. [Figure 12] This is a diagram showing an example of association information in which a meeting ID and device identification information are associated and stored in the association information storage unit. [Figure 13] This is a diagram showing an example of account information stored in the account information storage unit. [Figure 14] This is an example of a functional block diagram that explains the functions of an electronic blackboard by dividing them into blocks. [Figure 15] This is a diagram showing an example of device identification information and the like stored in the device information storage unit. [Figure 16] This is a diagram explaining the object information stored in the object information storage unit. [Figure 17] This is an example of a sequence diagram explaining the process of a user's handwritten stroke on an electronic blackboard being displayed together with an operator image on a terminal device at another location. [Figure 18] This is a diagram showing an example of an object and an operator image displayed on a terminal device at another location. [Figure 19] This is a diagram showing an example of a meeting registration screen displayed on a terminal device. [Figure 20] This is an example of a sequence diagram explaining the process of a user registering a meeting device and an electronic blackboard in an information processing system. [Figure 21] This is an example of a sequence diagram explaining the process of a user registering a meeting device and an electronic blackboard in an information processing system when no meeting information has been created. [Figure 22] This is an example of a sequence diagram explaining the communication method between a terminal device and an electronic blackboard when the terminal device and the electronic blackboard are connected to the same network. [Figure 23] This is a diagram explaining the operation method of setting the direction of an electronic blackboard by pressing a position registration button. [Figure 24] This is a diagram showing a screen for confirming the direction set by a user. [Figure 25]This is an example of a sequence diagram explaining the process by which an information recording app determines the orientation of an electronic blackboard based on specific video or audio. [Figure 26] This is a diagram showing an example of a two-dimensional code displayed by the electronic blackboard as a specific video. [Figure 27] This is a diagram explaining a method for determining the orientation based on the two-dimensional code. [Figure 28] This is a diagram explaining a method for determining the orientation of the electronic blackboard when the electronic blackboard outputs a specific sound. [Figure 29] This is a diagram explaining the orientation for determining the orientation of the electronic blackboard based on the electronic blackboard detected by image processing such as machine learning. <00001​​​​​​​​​​​​​​​​​​​​​​​​This is an example of a flowchart illustrating the process by which a terminal device at another location that receives an operator image displays the operator image. [Figure 39] This is an example of a diagram showing a mark indicating that an operator's image is attached to the stroke displayed by a terminal device at another location. [Figure 40] This diagram illustrates the input and display area. [Modes for carrying out the invention]

[0009] The following describes an example of an embodiment for carrying out the present invention, including a device system and an information processing method performed by the device system.

[0010] <An example of how to create meeting minutes for remote meetings> Information about the operators who manipulated the objects displayed on the interactive whiteboards was not being shared. For example, if the information processing system shared object data displayed on the interactive whiteboard at one location with other locations in real time, users at other locations would be able to see who wrote what. Also, if the information processing system associated operators with objects, it would become clear who entered the objects, improving the quality of meeting minutes.

[0011] First, with reference to Figure 1, we will outline the method for creating meeting minutes using panoramic images and application screens. Figure 1 is a diagram illustrating the outline of creating recording information that saves application screens executed during a remote meeting along with panoramic images of the surroundings. As shown in Figure 1, a user at the local site 102 is conducting a remote meeting with another site 101 using the remote conferencing service system 90.

[0012] The recording information creation system 100 of this embodiment creates recording information (such as meeting minutes) using a horizontal panoramic image (hereinafter referred to as the panoramic image) acquired by processing information captured by a meeting device 60 equipped with an imaging means capable of capturing images of the surroundings in a 360° range, a microphone, and a speaker, and a screen created by an application (hereinafter referred to as the app) executed by the terminal device 10. For audio, the recording information creation system 100 combines the audio received by the remote conferencing app 42 and the audio acquired by the meeting device 60 and includes them in the recording information. The outline is described below.

[0013] (1) The terminal device 10 is running the information recording application 41 and the remote conferencing application, which will be described later. Other applications, such as one for displaying documents, may also be running. The information recording application 41 transmits the audio output by the terminal device 10 (including audio received by the remote conferencing application from other locations) to the meeting device 60. The meeting device 60 mixes (combines) the audio it has acquired with the audio from the remote conferencing application.

[0014] (2) The meeting device 60 is equipped with a microphone and, based on the direction from which the sound is acquired, processes the image to extract the speaker from the panoramic image and creates a speaker image. The meeting device 60 transmits both the panoramic image and the speaker image to the terminal device 10.

[0015] (3) The information recording application 41 running on the terminal device 10 can display the panoramic image 203 and the speaker image 204. The information recording application 41 combines the panoramic image 203 and the speaker image 204 with any application screen selected by the user (for example, the screen 103 of the remote conferencing application). For example, the panoramic image 203, the speaker image 204, and the application screen 103 are combined so that the panoramic image 203 and the speaker image 204 are on the left and the screen 103 of the remote conferencing application is on the right (hereinafter referred to as the combined image 105). The application screen is an example of screen information (described later) displayed by each application such as the remote conferencing application. Since the process in (3) is executed repeatedly, the combined image 105 becomes a video (hereinafter referred to as the combined image video). The information recording application 41 also combines the audio synthesized with the combined image video to create a video with audio.

[0016] In this embodiment, an example is described in which the panoramic image 203, speaker image 204, and application screen 103 are combined. However, the information recording application 41 may save these separately and arrange them on the screen during playback.

[0017] (4) The information recording application 41 accepts editing work (cutting out unnecessary parts by the user) and completes the combined image video. The combined image video constitutes part of the recorded information.

[0018] (5) The information recording application 41 sends the created combined image video (with audio) to the storage service system 70 for storage.

[0019] (6) The information recording application 41 also extracts only the audio from the combined image video (it may also keep the audio before combining) and sends the extracted audio to the information processing system 50. The information processing system 50 sends the audio to the speech recognition service system 80, which converts the audio into text data. The text data also includes data on how many minutes after the start of recording the speaker spoke.

[0020] In the case of real-time text conversion, the meeting device 60 directly transmits audio to the information processing system 50. The meeting device 60 then transmits the text data to the information recording application 41 in real time.

[0021] (7) The information processing system 50 additionally stores text data in the storage service system 70 which stores the combined image video. The text data constitutes part of the recorded information.

[0022] Furthermore, the information processing system 50 can perform billing processing for users based on the services they have used. For example, billing is calculated based on the amount of text data, the file size of combined images and videos, processing time, etc.

[0023] In this way, the combined image video displays panoramic images of the surroundings including the user, as well as images of the speaker, and also shows the screen of the application used during the remote meeting, such as the remote conferencing application 42. When participants or non-participants of the remote meeting view the combined image video as meeting minutes, the atmosphere of the remote meeting is recreated with a sense of presence. [Examples]

[0024] <Outline of the process for associating and displaying the operator's image with an object> Next, referring to Figure 2, we will outline the process by which terminal devices 8 at other locations display objects in association with operator images. Figure 2 is a diagram illustrating the outline of the process by which terminal devices 8 at other locations (an example of the first terminal device) display objects displayed by the electronic whiteboard 2 together with operator images 7. Note that in Figure 2, at least the electronic whiteboard 2 and the terminal devices 8 at other locations are participating in the same meeting, and the terminal devices 8 at other locations share the objects displayed by the electronic whiteboard 2 with the electronic whiteboard 2.

[0025] (1) The meeting device 60 repeatedly captures images of the surroundings and transmits panoramic images and speaker images to the terminal device 10 (an example of a second terminal device). The panoramic image is assumed to include the operator 121.

[0026] (2) When the electronic whiteboard 2 is operated, the electronic whiteboard 2 transmits an operation start (request to extract operator image 7) to the meeting device 60 via the terminal device 10. When the meeting device 60 is notified of the operation start (request to extract operator image 7), it extracts the electronic whiteboard 2 from the panoramic image. The panoramic image may be captured in response to the start of operation. Since the electronic whiteboard 2 was operated at that time, the operator is visible in the extracted image. The extracted image is called operator image 7.

[0027] (3) The terminal device 10 transmits the operator image 7 to the electronic whiteboard 2.

[0028] (4) The electronic whiteboard 2 transmits, for example, the operator image 7 and stroke data to the information processing system 50 after each stroke. If strokes are being written continuously (if the pen-down for the next stroke is detected before a certain amount of time has elapsed after the pen-up), the electronic whiteboard 2 does not need to transmit the operator image 7.

[0029] (5) The information processing system 50 assigns an object ID to the stroke data and assigns identification information to the operator image 7, and associates the stroke data with the identification information of the operator image 7. The information processing system 50 also transmits the stroke data and the identification information of the operator image 7 to the terminal device 8 at another location. The information processing system 50 may also transmit the stroke data and the operator image 7 itself to the terminal device 8 at another location.

[0030] (6) Terminal devices 8 at other locations receive stroke data and identification information of the operator image 7, specify the identification information of the operator image 7, and request the operator image 7 from the information processing system 50.

[0031] (7) In response to this request, the information processing system 50 transmits the operator image 7, which is associated with the identification information of the operator image 7, to the terminal device 8 at another location.

[0032] (8) When a terminal device 8 at another location receives the operator image 7, it displays the operator image 7 in correspondence with the stroke 120.

[0033] As shown in Figure 2, the operator image 7 is displayed near the stroke at other locations, making it easy for users at other locations to understand who is writing (or has written) it. In addition, the information processing system 50 associates the operator image 7 with the object data, making it clear who entered the object and improving the content of the meeting minutes.

[0034] <About Terminology> An operator image is an image that includes the operator operating the electronic whiteboard 2. Since the electronic whiteboard 2 may be included along with the operator, the operator image may also be called an equipment image.

[0035] A stroke is a series of operations in which a user presses an input device against a display, moves it continuously, and then lifts it from the display. A stroke includes tracking the user's movement without contact with the display. In this case, the display device may initiate a stroke using, for example, a mouse or pointing device, a user's gesture, pressing a button with the user's hand or foot, or other means. Furthermore, the user may end a stroke using the same or different gesture, releasing a button, or using a mouse or pointing device.

[0036] Stroke data is information displayed on the screen based on the trajectory of coordinates entered by an input means. Stroke data may be interpolated as appropriate. Handwritten data is data that has one or more stroke data. Handwritten input indicates that handwritten data is entered by the user. Handwritten input may be performed using a touch interface, a tactile object such as a pen or stylus, or the user's body. Handwritten input may also be performed via other types of input, such as gesture-based input, hand movement tracking input, or other touch-free input by the user. While embodiments of the present invention refer to handwritten input and handwritten input data, other forms of handwritten input may be used.

[0037] The displayed items on the screen as a result of stroke data are called objects. While "object" generally means a target, in this embodiment it refers to the object being displayed. Objects converted from handwritten recognition of stroke data may include not only text, but also stamps displayed as fixed characters or marks such as "Done," shapes such as circles and stars, and straight lines. Text is primarily a string of characters (character codes) containing one or more characters, but also includes numbers and symbols. Text is sometimes referred to as a string of characters.

[0038] The input / display device can be any device that displays information. In this embodiment, the input / display device is described as an electronic blackboard. An electronic blackboard may also be called an electronic whiteboard, electronic information board, etc. A projector is known as a device equivalent to electronic blackboard 2. In addition, the input / display device may be a digital signage, television, display, video conferencing terminal, etc.

[0039] An application (app) is software developed and used for a specific function or purpose. Applications can be native apps or web apps. A web app (a cloud-based app provided by a cloud service) may also work in conjunction with a native app or web browser.

[0040] A running app refers to an app in its state from the time it is launched until it is closed. The app does not need to be active (the app in the foreground); it only needs to be running in the background.

[0041] Recorded information refers to information recorded by the information recording application 41, which is stored and made viewable as information linked to the identification information of a particular meeting, and includes, for example, the following information: Video information created based on screen information displayed by the selected application (such as a remote conferencing application), image information of the device's surroundings acquired by the device, etc. • Audio information acquired and synthesized during a meeting using a remote conferencing application (terminal device) and a meeting device at the local office. • Text information obtained by converting the acquired audio into text. Other data and images related to the meeting. For example, document files used during the meeting, additional notes, translated text data, images and stroke data created using a cloud-based electronic whiteboard service during the meeting, etc. For example, if the information recording application 41 records the screen of a remote meeting application or the proceedings of a meeting at a local office, the recorded information may become the minutes of the meeting that took place. The minutes are just one example of the recorded information, and the name of the recorded information changes depending on the content of the remote meeting or what was done at the local office. For example, it could be called a record of communication, a record of the local office's status, etc. The recorded information also includes multiple file formats, such as video files (combined video images, etc.), audio files, text data (text data from speech recognition), document files, image files, and tabular files. Since the files are related to each other in relation to the meeting identification information, they can be viewed chronologically, either all at once or selectively.

[0042] A tenant is a group of users (companies, local governments, and some of these organizations) that have contracted to receive services from a service provider. The creation of record information and its conversion to text data in this embodiment are performed because the tenant has a contract with the service provider.

[0043] Remote communication refers to communication with someone located at a physically distant location, using software and terminal devices to communicate via voice and video. One example of remote communication is a remote conference, which may also be called a meeting, discussion, consultation, contract application, gathering, assembly, seminar, workshop, study group, seminar, training session, etc.

[0044] A "base" refers to a location where various activities are carried out using network-connected devices. An example of a base is a conference room. A conference room is a room set up primarily for the purpose of holding meetings. A base can also be a home, reception area, store, warehouse, or outdoor work site, or any other location or space where terminals and devices can be installed.

[0045] Speech refers to human speech sounds and ambient sounds, while speech data is the data representation of speech. However, in this embodiment, we will not strictly distinguish between the two. Furthermore, "other locations" refer to different places located away from the local location, such as a different building or a different conference room.

[0046] <Example System Configuration> Next, the system configuration of the recording information creation system 100 will be explained with reference to Figure 3. Figure 3 shows an example configuration of the recording information creation system 100. In Figure 3, one of several locations where remote meetings are held (local location 102) is shown, and the terminal device 10 at local location 102 communicates with the information processing system 50, the storage service system 70, and the remote meeting service system 90 via the network. A meeting device 60 and an electronic whiteboard 2 are also located at local location 102, and the terminal device 10 is connected to the meeting device 60 via a USB cable or the like for communication. The terminal device 10 may also communicate with the meeting device 60 via a LAN.

[0047] Terminal device 10 operates at least an information recording application 41 and a remote conferencing application 42. The remote conferencing application 42 can communicate with terminal devices 10 at other locations 101 via a remote conferencing service system 90 on the network, allowing users at each location to hold meetings remotely. The information recording application 41 creates recording information for remote meetings conducted by the remote conferencing application 42 using the functions of the information processing system 50 and the meeting device 60.

[0048] In this embodiment, we describe an example of creating recording information during a remote meeting, but the meeting does not have to be one that communicates with remote locations. In other words, the meeting may be one in which only participants from a single location are present. In this case, only the audio collected by the meeting device 60 is saved without synthesis, and there is no change in the processing of the information recording application 41.

[0049] The terminal device 10 has a built-in (or external) camera with a normal field of view that captures an image of the front, including the user 107 operating the terminal device 10. A normal field of view is an image that is not a panoramic image, but in this embodiment, it is mainly a planar image that is not curved like a 360-degree image. The terminal device 10 also has a built-in (or external) microphone that collects ambient sounds such as those of the user operating the terminal device 10. Therefore, the user can conduct a conventional remote conference using the remote conference application 42 without being aware of the information recording application 41. The information recording application 41 and the meeting device 60 do not affect the remote conference application 42, except for the increased processing load on the terminal device 10.

[0050] The information recording application 41 is an application that records information by communicating with the meeting device 60 and creating recorded information. The meeting device 60 is a meeting device equipped with a panoramic image capture device, a microphone, and a speaker. The camera on the terminal device 10 can only capture a limited area in front of it, but the meeting device 60 can capture the entire surrounding area (not necessarily all around) of the meeting device 60. The meeting device 60 can always keep multiple participants 106, as shown in Figure 3, within its field of view.

[0051] In addition, the meeting device 60 extracts speaker images from the panoramic image, and combines the audio acquired by the meeting device 60 with the audio output by the terminal device 10 (including the audio received by the remote conferencing application 42). The meeting device 60 is not limited to being placed on a table or stand; it can be placed anywhere within the local site 102. Since the meeting device 60 can capture a 360-degree image, it can be placed on the ceiling, for example. The meeting device 60 can also be installed at other sites or at any of the sites.

[0052] The information recording application 41 displays a list of applications running on the terminal device 10, performs image synthesis (creation of combined image videos) for the recorded information described above, plays the combined image videos, accepts editing requests, etc. The information recording application 41 also displays a list of remote meetings that have been held or are scheduled to be held. The list of remote meetings is used for information related to the recorded information, allowing the user to link remote meetings with the recorded information.

[0053] The remote conferencing application 42 is an application that enables a terminal device to communicate remotely with other terminal devices at other locations 101 through communication connections, transmission and reception of images and audio, display of images, and output of audio. The remote conferencing application can also be called a remote communication application or a remote information sharing application.

[0054] The information recording application 41 and the remote conferencing application 42 may be either web applications or native applications. A web application is an application in which a program on a web server and a program on a web browser or a native application work together to process data, and does not require installation on the terminal device 10. A native application is an application that is installed on the terminal device 10 and used by it. In this embodiment, both will be described as native applications.

[0055] Terminal device 10 may be a general-purpose information processing device equipped with communication functions, such as a PC (Personal Computer), smartphone, or tablet device. Terminal device 10 may also be an electronic whiteboard 2, game console, PDA (Personal Digital Assistant), wearable PC, car navigation system, industrial machinery, medical equipment, network-connected home appliances, etc. Terminal device 10 only needs to be a device on which at least the information recording application 41 and the remote conferencing application 42 can run.

[0056] The electronic whiteboard 2 displays data handwritten on a touch panel using input methods such as a pen or finger. The electronic whiteboard 2 can communicate with terminal devices 10 etc. via wired or wireless connection and can capture the screen displayed by terminal devices 10 and display it on the display. The electronic whiteboard 2 can convert handwritten data into text data and share information displayed on the display with electronic whiteboards 2 at other locations. The electronic whiteboard 2 may also be a simple whiteboard without a touch panel on which a projector projects images. Alternatively, the electronic whiteboard 2 may be a tablet device, notebook PC (Personal Computer), PDA (Personal Digital Assistant), game console, etc., equipped with a touch panel.

[0057] The electronic whiteboard 2 can communicate with the information processing system 50. For example, after being powered on, the electronic whiteboard 2 can receive information from the information processing system 50 by polling it. The electronic whiteboard 2 and the meeting device 60, which can capture images of the electronic whiteboard 2, operate as a display system.

[0058] The information processing system 50 consists of one or more information processing devices located on a network. The information processing system 50 has one or more server applications that perform processing in cooperation with the information recording application 41, and infrastructure services. These server applications manage the list of remote meetings, manage recorded information from remote meetings, and manage various settings and storage paths. The infrastructure services handle user authentication, contracts, billing, etc. The meeting device 60, the electronic whiteboard 2, and the information processing system 50 function as a device system.

[0059] Furthermore, all or part of the functions of the information processing system 50 may reside in a cloud environment or in an on-premises environment. The information processing system 50 may consist of multiple server devices or a single information processing device. For example, server applications and infrastructure services may be provided by separate information processing devices, and furthermore, there may be separate information processing devices for each function within the server application. The information processing system 50 may be integrated with the storage service system 70 and the speech recognition service system 80 described below.

[0060] The storage service system 70 is a storage means on a network and provides a storage service that accepts the storage of files, etc. Examples of storage service systems 70 include OneDrive®, Google Workspace®, and Dropbox®. The storage service system 70 may also be an on-premises NAS (Network Attached Storage).

[0061] The speech recognition service system 80 provides a service that performs speech recognition on voice data and converts it into text data. The speech recognition service system 80 may be a general-purpose commercial service or it may be part of the functions of the information processing system 50. In addition, the speech recognition service system 80 may be configured and used differently for each user, tenant, or meeting.

[0062] <Example Hardware Configuration> Referring to Figure 4, the hardware configuration of the information processing system 50 and terminal device 10 according to this embodiment will be described.

[0063] <<Information Processing Systems and Terminal Devices>> Figure 4 shows the hardware configuration of an example of the information processing system 50 and terminal device 10 according to this embodiment. As shown in Figure 4, the information processing system 50 and terminal device 10 are built using a computer and include a CPU 501, ROM 502, RAM 503, HD (Hard Disk) 504, HDD (Hard Disk Drive) controller 505, display 506, external device connection I / F (Interface) 508, network I / F 509, bus line 510, keyboard 511, pointing device 512, optical drive 514, and media I / F 516.

[0064] Of these, the CPU 501 controls the operation of the entire information processing system 50 and terminal device 10. The ROM 502 stores programs used to drive the CPU 501, such as IPL. The RAM 503 is used as the work area for the CPU 501. The HD 504 stores various data such as programs. The HDD controller 505 controls the reading or writing of various data to the HD 504 according to the control of the CPU 501. The display 506 displays various information such as cursors, menus, windows, characters, or images. The external device connection I / F 508 is an interface for connecting various external devices. In this case, external devices include, for example, USB (Universal Serial Bus) memory and printers. The network I / F 509 is an interface for data communication using a network. The bus line 510 is an address bus, data bus, etc., for electrically connecting each component such as the CPU 501 shown in Figure 4.

[0065] The keyboard 511 is a type of input means equipped with multiple keys used for inputting characters, numbers, or various instructions. The pointing device 512 is a type of input means used for selecting and executing various instructions, selecting processing targets, moving the cursor, etc. The optical drive 514 controls the reading or writing of various data to the optical storage medium 513, which is an example of a removable recording medium. The optical storage medium 513 may be a CD, DVD, Blu-ray (registered trademark), etc. The media I / F 516 controls the reading or writing (storage) of data to the recording medium 515, such as flash memory.

[0066] <<Meeting Devices>> The hardware configuration of the meeting device 60 will be explained using Figure 5. Figure 5 is an example of a hardware configuration diagram for a meeting device 60 capable of capturing 360° video. In the following, the meeting device 60 is assumed to be a device that uses an image sensor to capture 360° video around the device at a predetermined height, but it may have one or more image sensors. Furthermore, it does not necessarily have to be a dedicated device; a 360° video capture unit can be attached to a PC, digital camera, smartphone, etc., to achieve substantially the same function.

[0067] As shown in Figure 5, the meeting device 60 consists of an imaging unit 601, an image processing unit 604, an imaging control unit 605, a microphone 608, a sound processing unit 609, a CPU (Central Processing Unit) 611, a ROM (Read Only Memory) 612, an SRAM (Static Random Access Memory) 613, a DRAM (Dynamic Random Access Memory) 614, an operation unit 615, an external device connection interface 616, a communication unit 617, an antenna 617a, a sound sensor 618, and a recessed terminal 621 for Micro USB.

[0068] Of these, the imaging unit 601 includes a wide-angle lens (a so-called fisheye lens) 602 with a 360° field of view for forming a hemispherical image, and an image sensor 603 provided corresponding to each wide-angle lens. The image sensor 603 has an image sensor such as a CMOS (Complementary Metal Oxide Semiconductor) sensor or a CCD (Charge Coupled Device) sensor that converts the optical image from the fisheye lens 602 into electrical signal image data and outputs it, a timing generation circuit that generates horizontal or vertical synchronization signals and pixel clocks for this image sensor, and a group of registers in which various commands and parameters necessary for the operation of this image sensor are set. The imaging unit 601 may also be a 360° camera, and is an example of an imaging means capable of imaging 360° around the meeting device 60.

[0069] Each image sensor 603 of the imaging unit 601 is connected to the image processing unit 604 via a parallel I / F bus. On the other hand, the image sensor 603 of the imaging unit 601 is connected to the imaging control unit 605 via a serial I / F bus (such as an I2C bus). The image processing unit 604, the imaging control unit 605, and the sound processing unit 609 are connected to the CPU 611 via bus 610. Furthermore, ROM 612, SRAM 613, DRAM 614, operation unit 615, external device connection I / F 616, communication unit 617, and sound sensor 618 are also connected to bus 610.

[0070] The image processing unit 604 receives image data output from the image sensor 603 via a parallel I / F bus, performs predetermined processing on each image data, and creates panoramic images and speaker images from the fisheye video. Furthermore, the image processing unit 604 combines the panoramic image and speaker images, etc., to output a single video.

[0071] The imaging control unit 605 generally uses the I2C bus to set commands and other information in the registers of the image sensor 603, with the imaging control unit 605 acting as the master device and the image sensor 603 as the slave device. It receives necessary commands and other information from the CPU 611. The imaging control unit 605 also uses the I2C bus to acquire status data and other information from the registers of the image sensor 603 and send it to the CPU 611.

[0072] Furthermore, the imaging control unit 605 instructs the image sensors 603a and 603b to output image data when the image capture start button on the operation unit 615 is pressed or when an image capture start instruction is received from the PC. Depending on the meeting device 60, there may also be functions to display a preview or video on a display (for example, the display of a PC or smartphone). In this case, the image data output from the image sensor 603 is performed continuously at a predetermined frame rate (frames / minute).

[0073] Furthermore, as will be described later, the imaging control unit 605 also functions as a synchronization control means that works in cooperation with the CPU 611 to synchronize the output timing of image data from the image sensor 603. In this embodiment, the meeting device 60 is not provided with a display, but a display unit may be provided.

[0074] Microphone 608 converts sound into sound (signal) data. The sound processing unit 609 receives the audio data output from microphone 608 through the I / F bus and performs predetermined processing on the audio data.

[0075] The CPU 611 controls the overall operation of the meeting device 60 and performs necessary processing. The ROM 612 stores various programs for the CPU 611. The SRAM 613 and DRAM 614 are work memories that store programs executed by the CPU 611 and data in progress. In particular, the DRAM 614 stores image data in progress and processed equirectangular projection image data from the image processing unit 604.

[0076] The control unit 615 is a collective term for the control buttons, such as the image capture start button 615a. By operating the control unit 615, the user can start imaging and recording, turn the power on / off, establish a communication connection, and input various settings such as imaging modes and imaging conditions.

[0077] The external device connection interface 616 is an interface for connecting various external devices. In this case, external devices include, for example, a PC (Personal Computer), a display, a projector, an electronic whiteboard, etc. The external device connection interface 616 may also be equipped with, for example, a USB terminal, an HDMI® terminal, etc. Video data and image data stored in the DRAM 614 are transmitted to an external terminal via this external device connection interface 616 or recorded on external media. Alternatively, multiple external device connection interfaces 616 may be used to transmit image information captured by the meeting device 60 to a PC via USB for recording, while simultaneously acquiring video (for example, screen information to be displayed in a remote conferencing application) from the PC to the meeting device 60, and then transmitting it from the meeting device 60 via HDMI to other external devices (display, projector, electronic whiteboard, etc.) for display.

[0078] The communication unit 617 may communicate with a cloud server via the internet using wireless communication technology such as Wi-Fi through the antenna 617a provided on the meeting device 60, and transmit the stored video data and image data to the cloud server. Alternatively, the communication unit 617 may communicate with nearby devices using short-range wireless communication technology such as BLE (Bluetooth Low Energy; registered trademark) or NFC.

[0079] The audio sensor 618 is a sensor that acquires 360° audio information in order to determine from which direction loud audio is being input within a 360° radius (horizontal plane) around the meeting device 60. The audio processing unit 609 identifies the direction of the strongest sound based on the input 360° audio parameters and outputs the audio input direction within the 360°.

[0080] Furthermore, other sensors (such as compass / accelerometers or GPS) can be used to calculate direction, position, angle, acceleration, etc., and these can be used for image correction or to add location information.

[0081] The image processing unit 604 also performs the following processing:

[0082] The CPU 611 creates a panoramic image in the following way: The CPU 611 performs predetermined camera image processing, such as Bayer conversion (RGB interpolation processing), on the RAW data input from the image sensor that inputs a spherical image to create a fisheye image (an image consisting of curved images). Furthermore, it performs planarization processing, such as DeWarp processing (distortion correction processing), on the created fisheye image (a curved image) to create a panoramic image (an image consisting of flat images) that captures 360° around the meeting device 60.

[0083] The CPU 611 creates speaker images in the following way: The CPU 611 creates speaker images by extracting the speaker from a panoramic image (flat image) that captures the surrounding 360°. The CPU 611 uses the direction of the audio input, identified from the 360° output by the audio sensor 618 and the audio processing unit 609, as the direction of the speaker, and extracts the speaker image from the panoramic image. In this case, the method for extracting a person's image from the direction of the audio input is to cut out 30° around the audio direction identified from the 360°, and then perform face detection within that area to extract the image. The CPU 611 further identifies speaker images for a specific number of people (e.g., 3 people) who have spoken most recently from the extracted speaker images.

[0084] The panoramic image and the images of one or more speakers may be sent individually to the information recording application 41, or the meeting device 60 may create a single image from them and send it to the information recording application 41. In this embodiment, the panoramic image and the images of one or more speakers are sent individually from the meeting device 60 to the information recording application 41.

[0085] Figure 6 illustrates the imaging range of the meeting device 60. As shown in Figure 6(a), the meeting device 60 images a 360° range in the horizontal direction. As shown in Figure 6(b), the meeting device 60 has an imaging range that is 0° horizontal to the height of the meeting device 60, and a predetermined angle above and below it.

[0086] Figure 7 illustrates the extraction of panoramic and speaker images. As shown in Figure 7, the image captured by the meeting device 60 forms a part 110 of a sphere and therefore has a three-dimensional shape. As shown in Figure 6(b), the meeting device 60 divides the field of view into predetermined angles vertically and horizontally and performs perspective projection transformation. By performing perspective projection transformation seamlessly across the entire 360° horizontal direction, a predetermined number of planar images are obtained, and by stitching these predetermined number of planar images together horizontally, a panoramic image 111 is obtained. The meeting device 60 also performs face detection from the panoramic image within a predetermined range centered on the direction of sound, and creates speaker images 112 by extracting 15° to the left and right (30° in total) from the center of the face.

[0087] <<Electronic Whiteboard>> Figure 8 is a hardware configuration diagram of the electronic whiteboard 2. As shown in Figure 8, the electronic whiteboard 2 is equipped with a CPU (Central Processing Unit) 401, ROM (Read Only Memory) 402, RAM (Random Access Memory) 403, SSD (Solid State Drive) 404, network I / F 405, and external device connection I / F (Interface) 406.

[0088] Of these, the CPU 401 controls the overall operation of the electronic whiteboard 2. The ROM 402 stores programs used to drive the CPU 401, such as the CPU 401 and the IPL (Initial Program Loader). The RAM 403 is used as the work area for the CPU 401. The SSD 404 stores various data, such as programs for the electronic whiteboard 2. The network I / F 405 controls communication with the communication network. The external device connection I / F 406 is an interface for connecting various external devices. In this case, external devices include, for example, a USB (Universal Serial Bus) memory 430 and external devices (microphone 440, speaker 450, camera 460).

[0089] Furthermore, the electronic whiteboard 2 is equipped with a capture device 411, a GPU 412, a display controller 413, a contact sensor 414, a sensor controller 415, an electronic pen controller 416, a short-range communication circuit 419, and an antenna 419a for the short-range communication circuit 419, a power switch 422, and selection switches 423.

[0090] Of these, the capture device 411 displays video information as a still image or video on the display of an external PC (Personal Computer) 470. The GPU (Graphics Processing Unit) 412 is a semiconductor chip specializing in graphics. The display controller 413 controls and manages screen display in order to output the output image from the GPU 412 to the display 480, etc. The contact sensor 414 detects when an electronic pen 490 or the user's hand H, etc., touches the display 480. The sensor controller 415 controls the processing of the contact sensor 414. The contact sensor 414 performs coordinate input and coordinate detection using an infrared blocking method. The method of this coordinate input and coordinate detection will be explained below. Two light-emitting and receiving devices installed at both ends of the upper side of the display 480 emit multiple infrared rays parallel to the display 480. Reflected by reflective members provided around the display 480, the two light-emitting and receiving devices receive the light that returns along the same optical path as the light emitted by the light-receiving element.

[0091] The contact sensor 414 outputs the infrared ID emitted by two light-receiving devices that are blocked by an object to the sensor controller 415, which then identifies the coordinate position of the object's contact. The electronic pen controller 416 communicates with the electronic pen 490 to determine whether the pen tip or the pen end has touched the display 480. The near-field communication circuit 419 is a communication circuit such as NFC (Near Field Communication) or Bluetooth (registered trademark). The power switch 422 is a switch for turning the power of the electronic whiteboard 2 ON / OFF. The selection switches 423 are a group of switches for adjusting, for example, the brightness and color of the display 480.

[0092] Furthermore, the electronic whiteboard 2 is equipped with a bus line 410. The bus line 410 is an address bus, data bus, etc., for electrically connecting each component, such as the CPU 401 shown in Figure 8.

[0093] Furthermore, the contact sensor 414 is not limited to an infrared blocking type; it may also be a capacitive type touch panel that identifies the contact position by detecting changes in capacitance. The contact sensor 414 may also be a resistive type touch panel that identifies the contact position by voltage changes between two opposing resistive films. The contact sensor 414 may also be an electromagnetic induction type touch panel that identifies the contact position by detecting electromagnetic induction caused by contact between an object and the display unit. In addition, the contact sensor 414 may use various other detection means. Moreover, the electronic pen controller 416 may determine whether there is a touch not only on the tip and end of the electronic pen 490, but also on the part of the electronic pen 490 held by the user and other parts of the electronic pen.

[0094] <About the features> Next, with reference to Figure 9, the functional configuration of the recording information creation system 100 will be described. Figure 9 is an example of a functional block diagram that explains the functions of the terminal device 10, meeting device 60, and information processing system 50 in the recording information creation system 100 by dividing them into blocks.

[0095] <<Terminal device>> The information recording application 41 operating on the terminal device 10 includes a communication unit 11, an operation reception unit 12, a display control unit 13, an application screen acquisition unit 14, an audio acquisition unit 15, a device communication unit 16, a recording control unit 17, an audio data processing unit 18, a recording playback unit 19, an upload unit 20, an editing processing unit 21, a device recognition unit 22, an analysis unit 23, an operator image transmission unit 24, and a cropping request unit 25. Each of these parts of the terminal device 10 is a function or means of functioning, realized by any of the components shown in Figure 4 operating according to instructions from the CPU 501 in accordance with the information recording application 41 deployed from the HD 504 to the RAM 503. The terminal device 10 also has a storage unit 1000 constructed by the HD 504, etc., shown in Figure 4. An information storage unit 1001 is constructed in the storage unit 1000.

[0096] The communication unit 11 communicates various types of information with the information processing system 50 via the network. For example, the communication unit 11 receives a list of remote meetings from the information processing system 50 and sends voice data recognition requests to the information processing system 50. The communication unit 11 also receives operation start and operation end notifications from the electronic whiteboard 2.

[0097] The display control unit 13 displays various screens that serve as the user interface in the information recording application 41 according to the screen transitions set in the information recording application 41. The operation reception unit 12 receives various operations on the information recording application 41.

[0098] The application screen acquisition unit 14 acquires screen information displayed by the application selected by the user, or screen information of the desktop screen, etc., from the OS (Operating System). If the application selected by the user is the remote conferencing application 42, the screen generated by the remote conferencing application 42 is obtained (images of terminal device users captured by terminal device cameras at each location, images of shared documents, images including participant icons and participant names, etc.). The screen information displayed by the application (application screen) is information displayed as a window by the running application and acquired as an image by the information recording application. The application window is drawn as an area of ​​the window within the entire desktop image and displayed on a monitor, etc. The screen information displayed by the application can be acquired by other applications (information recording applications, etc.) as an image file or a video file composed of a series of images via the OS (Operating System) API or the API of the application displaying it. Similarly, the screen information of the desktop screen is information composed of images of the desktop screen generated by the OS, and can be acquired as an image file or video file via the OS API. The format of these image files may be bitmap, PNG, or other formats. The video file format can be MP4 or other formats.

[0099] The audio acquisition unit 15 acquires audio output from the microphone or earphones of the terminal device 10 (including audio data received from the remote conferencing application 42 during a remote conference). The audio acquisition unit 15 can acquire audio even if the output audio is muted. The user does not need to perform any operations such as selecting the remote conferencing application 42 regarding audio data; the audio acquisition unit 15 can acquire the audio that the terminal device 10 can output via the OS or application's API (Application Interface). This also allows the audio data received by the remote conferencing application 42 from other locations 101 to be acquired. If the remote conferencing application 42 is not running or a remote conference is not in progress, the information recording application 41 may not be able to acquire audio data. Note that the audio acquired by the audio acquisition unit 15 may not include the audio collected by the terminal device 10, but only the output audio data. This is because the meeting device 60 separately collects audio from the locations.

[0100] The device communication unit 16 communicates with the meeting device 60 using a USB cable or the like. The device communication unit 16 may communicate with the meeting device 60 using wireless LAN, Bluetooth (registered trademark), etc. The device communication unit 16 receives panoramic images and speaker images from the meeting device 60 and transmits the audio data acquired by the audio acquisition unit 15 to the meeting device 60. The device communication unit 16 receives the audio data synthesized by the meeting device 60.

[0101] The recording control unit 17 combines the panoramic image and speaker image received by the device communication unit 16, and the application screen acquired by the application screen acquisition unit 14, to create a combined image. The recording control unit 17 also repeatedly connects the created combined images in chronological order to create a combined image video, and combines the synthesized audio data with the combined image video to create a combined image video with audio. The combination of the panoramic image and speaker image may be performed by the meeting device 60. Alternatively, videos consisting of each image, such as the panoramic image, speaker image, application screen, and images consisting of the panoramic image and speaker image, may be stored as separate video files in the storage service system 70. In that case, the panoramic video, speaker video, application screen video, and combined video of the panoramic image and speaker image may be called up and displayed on a single display screen when viewed.

[0102] The audio data processing unit 18 either extracts the audio data combined into the combined image video, or requests the information processing system 50 to convert the synthesized audio data received from the meeting device 60 into text data.

[0103] The recording and playback unit 19 plays back the combined image video. The combined image video is stored in the terminal device 10 during recording and then uploaded to the information processing system 50.

[0104] When the remote conference ends, the upload unit 20 transmits the combined image and video to the information processing system 50.

[0105] The editing processing unit 21 performs editing of the combined image video (such as deleting parts or splicing them together) in response to user operations.

[0106] The device recognition unit 22 learns the shape (circumscribed rectangle) of the electronic whiteboard 2 using machine learning and detects the electronic whiteboard 2 from the panoramic image. The device recognition unit 22 may also recognize the device simply by pattern matching without using machine learning. The meeting device 60 may perform the processing that the device recognition unit 22 performs.

[0107] The analysis unit 23 detects the two-dimensional code contained in the panoramic image and analyzes the two-dimensional code to obtain information contained in the code, such as the device identification information of the electronic whiteboard 2. The meeting device 60 may perform the processing that the analysis unit 23 performs.

[0108] The operator image transmission unit 24 transmits the operator image 7 sent from the meeting device 60 to the information processing system 50. The information processing system 50 transmits the operator image 7 to terminal devices 8 at other locations participating in the same meeting. The meeting device 60 may perform the processing that the operator image transmission unit 24 performs.

[0109] When the cropping request unit 25 receives notification of the start of an operation from the electronic whiteboard 2, it requests the meeting device 60 to crop the operator image 7. Furthermore, when the cropping request unit 25 receives notification of the end of an operation from the electronic whiteboard 2, it requests the meeting device 60 to finish cropping the operator image 7.

[0110] Figure 10 shows the video recording information stored in the information storage unit 1001. The video recording information includes items such as meeting ID, recording ID, update date and time, title, upload, and storage location. When a user logs in to the information processing system 50, the information recording application 41 downloads meeting information from the meeting information storage unit 5001 of the information processing system 50. The meeting ID and other information included in the meeting information are reflected in the video recording information. The video recording information in Figure 10 is held by a terminal device 10 operated by a particular user.

[0111] The meeting ID is identification information that identifies a remote meeting that has been held. The meeting ID is assigned when the remote meeting is scheduled in the meeting management system 9, or it is assigned by the information processing system 50 upon request from the information recording application 41. The meeting management system 9 is a system for registering meeting and remote meeting schedules, URLs (meeting links) for starting remote meetings, and reservation information for equipment used in meetings. It is a scheduler, etc., that is connected to from the terminal device 10 via the network. The meeting management system 9 can also transmit the registered schedules, etc., to the information processing system 50.

[0112] The recording ID is identification information that identifies the combined video image recorded during a remote meeting. The recording ID is assigned by the meeting device 60, but it may also be assigned by the information recording application 41 or the information processing system 50. The same meeting ID may be assigned to a different recording ID if the recording of the remote meeting was stopped midway through but resumed for some reason.

[0113] • The update date and time is the date and time when the combined image / video was updated (recording ended). If the combined image / video was edited, it will be the date and time of editing.

[0114] The title is the name of the meeting. This may be set when registering the meeting in the meeting management system 9, or it may be set arbitrarily by the user.

[0115] • Upload indicates whether or not the combined image video has been uploaded to the information processing system 50.

[0116] The storage location indicates the location (URL or file path) where the combined image / video and text data are stored in the storage service system 70. Therefore, users can view the uploaded combined image / video at their discretion. Note that the combined image / video and text data are saved with separate filenames, for example, following the URL.

[0117] • Terminal devices at other locations 8 Regarding the terminal devices 8 at other locations, the function of sharing objects displayed by the electronic whiteboard 2 will be described. The terminal devices 8 at other locations may have the same functions as the terminal device 10 at the local location, in addition to those shown in the figure. The terminal devices 8 at other locations have a communication unit 71, an operation reception unit 72, a display control unit 73, and an operator image acquisition unit 74. Each of these parts of the terminal device 8 at other locations is a function or means of functioning, which is realized by any of the components shown in Figure 4 operating according to instructions from the CPU 501 in accordance with a program (electronic whiteboard 2 application or web browser) deployed from HD 504 to RAM 503.

[0118] The communication unit 71 communicates with the information processing system 50 to receive various information, including object data displayed on the electronic whiteboard 2 and operator images 7. The communication unit 71 also transmits handwritten object data from terminal devices 8 at other locations to the information processing system 50.

[0119] The operation reception unit 72 receives various operations for terminal devices 8 at other locations. For example, the operation reception unit 72 may receive stroke inputs for the electronic whiteboard 2.

[0120] The display control unit 73 displays object data and handwritten strokes received by the communication unit 71 on the display of a terminal device 8 at another location.

[0121] If the operator image acquisition unit 74 has identification information for the operator image 7 attached to the object data, it acquires the operator image 7 from the information processing system 50 by specifying the identification information for the operator image 7 via the communication unit 71.

[0122] <<Meeting Devices>> Let's return to Figure 9 for explanation. The meeting device 60 includes a terminal communication unit 61, a panoramic image generation unit 62, a speaker image generation unit 63, a sound collection unit 64, a speech synthesis unit 65, an operator image generation unit 66, and a device direction detection unit 67. Each of these parts of the meeting device 60 is a function or means of functioning, realized by any of the components shown in Figure 5 operating according to instructions from the CPU 611 in accordance with a program expanded from the ROM 612 to the DRAM 614.

[0123] The terminal communication unit 61 communicates with the terminal device 10 using a USB cable or the like. The terminal communication unit 61 does not only need to be connected by a wired cable, but it is also acceptable if it can communicate with the terminal device 10 via wireless LAN, Bluetooth (registered trademark), etc.

[0124] The panoramic image generation unit 62 creates a panoramic image. The speaker image generation unit 63 creates a speaker image. The methods for creating these images are explained in Figures 6 and 7. The panoramic image generation unit 62 also serves as an image data acquisition unit.

[0125] The sound collection unit 64 converts the audio signal acquired by the microphone of the meeting device 60 into audio data (digital). This allows the content spoken by users and participants at the terminal device 10 to be collected.

[0126] The speech synthesis unit 65 synthesizes the speech transmitted from the terminal device 10 with the speech collected by the sound collection unit 64. This combines speech spoken at other locations 101 with speech spoken at the local location 102.

[0127] The operator image generation unit 66 starts extracting the operator image 7 from the panoramic image in response to the start of an operation (extraction request) from the terminal device 10, and stops extracting the operator image 7 in response to the end of the operation (extraction termination request).

[0128] The device direction detection unit 67 detects a specific sound (sound of a specific frequency) to determine the direction of the electronic whiteboard 2 in the panoramic image. Alternatively, the device direction detection unit 67 may detect the direction of the electronic whiteboard 2 in the panoramic image based on a preset direction, detection of a two-dimensional code, or the shape of the electronic whiteboard 2. In this embodiment, the terminal device 10 is responsible for detecting the direction of the electronic whiteboard 2 based on user settings, detection of a two-dimensional code, or the shape of the electronic whiteboard 2. In this case, the device direction detection unit 67 receives the direction of the electronic whiteboard 2 from the terminal device 10.

[0129] <<Information Processing System>> The information processing system 50 includes a communication unit 51, an authentication unit 52, a screen generation unit 53, a communication management unit 54, an equipment management unit 55, a text conversion unit 56, and an object sharing unit 57. Each of these units in the information processing system 50 is a function or means of functioning, realized by any of the components shown in Figure 4 operating according to instructions from the CPU 501 following a program deployed from the HD 504 to the RAM 503. The information processing system 50 also has a storage unit 5000 constructed from the HD 504 and other components shown in Figure 4. The storage unit 5000 includes a conference information storage unit 5001, a recording information storage unit 5002, a mapping information storage unit 5003, an account information storage unit 5004, and an object data storage unit 5005. The object data storage unit 5005 will be explained in the description of the electronic whiteboard 2.

[0130] The communication unit 51 sends and receives various types of information with the terminal device 10. For example, the communication unit 51 may send a list of remote meetings to the terminal device 10 or receive a voice data recognition request from the terminal device 10.

[0131] The authentication unit 52 authenticates the user operating the terminal device 10. The authentication unit 52 authenticates the user, for example, by checking whether the authentication information (user ID and password) included in the authentication request received by the communication unit 51 matches the authentication information it has previously stored. The authentication information may be an IC card number, biometric authentication information such as a face or fingerprint, etc. The authentication unit 52 may also authenticate using an external authentication system or an authentication method such as OAuth.

[0132] The screen generation unit 53 generates screen information to be displayed by the terminal device 10. When the terminal device 10 runs a native application, the screen information is held by the terminal device 10, and the information to be displayed is transmitted in XML or the like. When the terminal device 10 runs a web application, the screen information is created using HTML, XML, CSS (Cascade Style Sheet), and JavaScript (registered trademark), etc.

[0133] The communication management unit 54 retrieves information about remote meetings from the meeting management system 9 using each user's account or a system account assigned to the information processing system 50. The communication management unit 54 stores the meeting information of scheduled meetings in the meeting information storage unit, associating it with the meeting ID. The communication management unit 54 also retrieves meeting information that users belonging to the tenant have viewing rights to. Since each meeting is assigned a meeting ID, the remote meeting and its recorded information are associated using the meeting ID.

[0134] When the equipment management unit 55 receives equipment identification information for the electronic whiteboard 2 from the terminal device 10, it associates the meeting ID with the electronic whiteboard 2 and meeting device 60 used in the meeting and stores the association information in the association information storage unit 5003. Combined image videos are also associated with the meeting ID, so handwritten data and combined image videos are also associated. Furthermore, when recording ends (when the meeting ends), the equipment management unit 55 deletes the association from the association information storage unit 5003.

[0135] The text conversion unit 56 converts the voice data requested by the terminal device 10 to be converted to text data into text data using an external voice recognition service. The text conversion unit 56 may also perform the conversion itself.

[0136] The object sharing unit 57 shares the input display area, object data, and operator image 7 between the terminal device 10 and the electronic whiteboard 2 that are participating in the same meeting. The object sharing unit 57 provides the input display area, allowing the electronic whiteboard 2 and the terminal device 10 to share the input display area, and accepts object input to the input display area on the electronic whiteboard 2 and the terminal device 10. The object sharing unit 57 can identify the meeting by attaching the meeting ID to the communication from the terminal device 10. In response to object input, the object sharing unit 57 transmits the object sent from one electronic whiteboard 2 to the other electronic whiteboard 2 (all devices participating in the meeting by linking to it), and vice versa. The object sharing unit 57 can also share objects and operator image 7 between the electronic whiteboard 2 and the terminal device 10. The transmitted content is not limited to objects only; the entire input display area in which the object is entered may also be transmitted and displayed.

[0137] The object sharing unit 57 may be, for example, an online whiteboard application or an online electronic blackboard application, which allows operators from multiple terminal devices 10 and electronic blackboards 2 to mutually write objects and create a blackboard online (on the cloud). Alternatively, the object sharing unit 57 may use an external web service of the information processing system 50. The object sharing unit 57 is just one example of a sharing unit.

[0138] Figure 11 shows an example of meeting information stored in the meeting information storage unit 5001, which is managed by the communication management unit 54. The communication management unit 54 can use the above account to obtain a list of remote meetings that the user belonging to the tenant has permission to view. Note that viewing permission may also be granted directly to the meeting information managed by the communication management unit 54 from the information recording application of the terminal device 10. Furthermore, the remote meeting information that a user belonging to the tenant has permission to view includes information on meetings created by the user and information on meetings that the user has been granted viewing permission for by other users. In this embodiment, remote meetings are used as an example, but the list of remote meetings also includes meetings held in only one conference room.

[0139] Meeting information is managed by a meeting ID, which is associated with the organizer ID, title (meeting name), start date and time, end date and time, electronic whiteboard 2, meeting device 60, etc. These are just examples of meeting information; meeting information may include other details as well.

[0140] The "Organizer ID" field indicates the organizer (host) of the meeting.

[0141] The title field should describe the content of the meeting, such as the meeting name or the agenda items from overseas.

[0142] The "Start Date and Time" field indicates the scheduled date and time when the meeting will begin.

[0143] The "End Date and Time" field indicates the scheduled end date and time of the meeting.

[0144] The item for Electronic Whiteboard 2 is the identification information for Electronic Whiteboard 2 associated with the meeting.

[0145] The item "Meeting Device 60" contains the identification information of the meeting device 60 used in the meeting.

[0146] As shown in Figures 10 and 11, the combined video footage recorded during the meeting is identified by the meeting ID.

[0147] The recording information stored in the recording information storage unit 5002 may be the same as in Figure 10. However, the information processing system 50 has a list of combined image videos recorded by all users belonging to the tenant. The storage location (path information such as the URL of the cloud storage system) may be entered by the user on the user settings screen of the information recording application 41 of the terminal device 10 and stored in the recording information storage unit 5002.

[0148] Figure 12 shows the mapping information stored in the mapping information storage unit 5003, which associates the meeting ID with the device identification information (electronic whiteboard 2 and meeting device 60). The mapping information is retained from the time the information recording application 41 transmits the device identification information to the information processing system 50 until the recording is terminated. Since the terminal device 10 is connected to the meeting device 60, the information processing system 50 can also associate and store the identification information (IP address, etc.) of the terminal device 10. Alternatively, if the terminal device 10 attaches the device identification information of the meeting device 60 to its communication, it can be identified as the terminal device 10 participating in the same meeting as the electronic whiteboard 2.

[0149] Figure 13 shows an example of account information stored in the account information storage unit 5004. The account information includes not only human users but also non-human users such as the electronic whiteboard 2 and the meeting device 60.

[0150] The user ID is identification information for users who can participate in the meeting, as well as for the electronic whiteboard 2 and meeting device 60.

[0151] • The type refers to the type of account for each user, electronic whiteboard 2, or meeting device 60.

[0152] • The names are the user's name, the names of the electronic whiteboard 2, and the meeting device 60.

[0153] • The email addresses are those of the user, electronic whiteboard 2, and meeting device 60, etc.

[0154] <<Electronic Whiteboard>> Figure 14 is a functional block diagram illustrating the functions of the electronic whiteboard 2 in blocks. The electronic whiteboard 2 includes a contact position detection unit 31, a drawing data generation unit 32, a data recording unit 33, a display control unit 34, a code generation unit 35, a communication unit 36, an audio data generation unit 37, an operation detection unit 38, a user detection unit 39, an object upload unit 40a, and an operator image receiving unit 40b. Each function of the electronic whiteboard 2 is realized by an instruction from the CPU 401, following a program deployed from the SSD 404 onto the RAM 403, in accordance with one of the components shown in Figure 8.

[0155] The contact position detection unit 31 detects the coordinates of the position where the electronic pen 490 makes contact with the contact sensor 414. The drawing data generation unit 32 obtains the coordinates of the contact point of the pen tip of the electronic pen 490 from the contact position detection unit 31. The drawing data generation unit 32 connects these coordinate points by interpolation to generate stroke data.

[0156] The display control unit 34 displays handwritten data, strings converted from handwritten data, and menus for user operation on the display.

[0157] The data recording unit 33 stores handwritten data written on the electronic whiteboard 2, shapes converted from the handwritten data into circles, triangles, etc., stamps such as "completed", PC screens, files, etc., in the object data storage unit 3002. Handwritten data, strings (including shapes), images such as PC screens, files, etc., are treated as objects. For handwritten data, a set of stroke data becomes a single object according to temporal divisions due to interruptions in handwriting and spatial divisions due to differences in handwriting location.

[0158] The communication unit 36 ​​connects to Wi-Fi or LAN and communicates with the information processing system 50. The communication unit 36 ​​transmits object data to the information processing system 50 and receives object data stored in the information processing system 50 and displays it on the display 480. The communication unit 36 ​​also transmits the start or end of an operation to the terminal device 10.

[0159] The code generation unit 35 generates a two-dimensional code by encoding the device identification information of the electronic whiteboard 2 stored in the device information storage unit 3001 and information indicating that it is a device usable in a meeting into a two-dimensional pattern. The code generation unit 35 can also encode the device identification information of the electronic whiteboard 2 and information indicating that it is a device usable in a meeting into a barcode. The device identification information may be a serial number or a UUID. The device identification information may also be set by the user.

[0160] The audio data generation unit 37 generates audio data by sampling signals of a preset frequency at regular intervals, similar to PCM conversion. The audio data is converted to analog by the D / A converter of the speaker 450 and output from the speaker 450.

[0161] The operation detection unit 38 detects user operations on the electronic whiteboard 2. For example, the operation detection unit 38 detects the start or end of an operation when the contact position detection unit 31 detects that the electronic pen 490 or the user's hand H has come into contact with the display 480 (touch panel) (even if it is just close).

[0162] The user detection unit 39 detects when a user approaches or moves away from the electronic whiteboard 2 using an infrared sensor, ultrasonic sensor, camera, etc.

[0163] The object upload unit 40a uploads the objects displayed by the electronic whiteboard 2 to the information processing system 50 at a predetermined time. This allows the objects to be shared between locations.

[0164] The operator image receiving unit 40b receives the operator image 7 from the terminal device 10. The operator image receiving unit 40b may receive the operator image 7 from the terminal device 10 via the information processing system 50, or it may receive it directly from the terminal device 10.

[0165] Furthermore, the electronic whiteboard 2 has a storage unit 3000 constructed by the SSD 404 shown in Figure 8. The storage unit 3000 comprises a device information storage unit 3001 and an object data storage unit 3002.

[0166] Figure 15 shows the device identification information and other data stored in the device information storage unit 3001. • The device identification information is the identification information for electronic whiteboard 2. The IP address is the IP address that other devices use to connect to the electronic whiteboard 2 via the network. • The password is used for authentication when other devices connect to the electronic whiteboard 2.

[0167] Figure 16 illustrates the object data stored in the object data storage unit 3002. The object data is information that manages the objects displayed by the electronic whiteboard 2. The object data is transmitted to the information processing system 50 and used as meeting minutes. The objects are associated with a meeting ID notified by the information processing system 50.

[0168] The Object ID field contains identifying information for an object. Objects include strokes, characters, shapes, images, etc. Characters are strings of characters (character codes) converted from handwritten data. Strings of characters are sometimes also called text data. Shapes are geometric shapes such as triangles and squares converted from handwritten data. Images are image data such as JPEG, PNG, PDF, and TIFF imported from a PC or the internet.

[0169] The data content items are a sequence of coordinate points that make up the stroke. The data content items are character codes, or data such as image files and their location information.

[0170] The input source field is the identification information of the device to which the object was entered. Handwriting on electronic whiteboard 2 registers the identification information of electronic whiteboard 2, while handwriting on terminal device 8 at another location registers the user ID (login user) who wrote the handwriting.

[0171] The identification information field for operator image 7 registers the identification information of operator image 7 associated with the object. Operator image 7 may be stored in the information processing system 50 or in the storage service system 70.

[0172] The save location is the URL or file path where the operator image 7 is saved.

[0173] <Action or process> Next, referring to Figure 17, we will explain the overall flow in which the meeting device 60 extracts the operator image 7 and the terminal device 8 at the other location displays the object and the operator image 7. Figure 17 is a sequence diagram illustrating the process in which the terminal device 8 at the other location displays strokes handwritten by the user on the electronic whiteboard 2 along with the operator image 7.

[0174] S1: Users participating in the meeting can write by hand using the electronic pen 490 or display their PC screen on the electronic whiteboard 2.

[0175] S2: The contact position detection unit 31 of the electronic whiteboard 2 detects the contact position of the electronic pen 490, and the operation detection unit 38 detects the start of operation. The user detection unit 39 may also detect the approach of a user. The communication unit 36 ​​of the electronic whiteboard 2 transmits a message to the terminal device 10 indicating that the electronic whiteboard 2 has been operated (that operation has been detected). The method of communication between the electronic whiteboard 2 and the terminal device 10 will be described later (Figures 19 to 22).

[0176] S3: The communication unit 11 of the information recording application 41 receives the start of an operation, so the cropping request unit 25 sends a request to crop the operator image 7 to the meeting device 60 via the device communication unit 16. The cropping request unit 25 may also send a request to crop the operator image 7 to the meeting device 60 in response to the user inputting an operation to crop the operator image.

[0177] S4: The terminal communication unit 61 of the meeting device 60 receives a cropping request, and the operator image generation unit 66 generates (crops) the operator image 7 from the panoramic image. The method for detecting the direction of the electronic whiteboard 2 will be described later. Note that the operator image 7 can also be cropped by the terminal device 10.

[0178] S5: The terminal communication unit 61 of the meeting device 60 transmits the requested operator image 7 to the terminal device 10. The device communication unit 16 of the terminal device 10 receives the operator image 7.

[0179] The meeting device 60 may generate an operator image 7 for each stroke, or for each object. Alternatively, the meeting device 60 may generate operator images 7 periodically.

[0180] S6: A user participating in the meeting finishes operating the interactive whiteboard 2. Operation ends when the pen is released or when there is inactivity for a certain period of time since the last operation.

[0181] S7: The operation detection unit 38 of the electronic whiteboard 2 detects the end of the operation. The end of the operation here is assumed to be the end of one stroke (pen up). The user detection unit 39 may also detect that the user has moved away. In response to the end of the operation, the operator image receiving unit 40b of the electronic whiteboard 2 requests the operator image 7 from the terminal device 10 via the communication unit 36. The communication unit 11 of the terminal device 10 transmits the operator image 7 to the electronic whiteboard 2.

[0182] S8: The communication unit 36 ​​of the electronic whiteboard 2 receives the operator image 7, and the object upload unit 40a transmits the operator image 7 to the information processing system 50 via the communication unit 36, specifying the meeting ID. The communication unit 51 of the information processing system 50 receives the operator image 7, and the object sharing unit 57 assigns an identification number to the operator image 7. The object sharing unit 57 stores the operator image 7 and its identification information in association. The identification information of the operator image 7 is transmitted to the terminal device 10.

[0183] S9: Next, the object upload unit 40a of the electronic whiteboard 2 transmits the object data (in this case, stroke data) and identification information of the operator image 7 to the information processing system 50 via the communication unit 36, specifying the conference ID.

[0184] S10: The communication unit 51 of the information processing system 50 receives the conference ID, object data, and identification information of the operator image 7, and the object sharing unit 57 assigns an object ID (the electronic whiteboard 2 may also assign the object ID). The object sharing unit 57 stores the object ID, object data, and identification information of the operator image 7 in association with each other. Note that the order of transmission of the object data and the operator image 7 may be reversed.

[0185] S11: When the information processing system 50 receives object data and operator image 7, the object sharing unit 57 identifies terminal devices 8 at other locations participating in the same conference using the conference ID, and the communication unit 51 transmits the identification information of the object data and operator image 7 to the terminal devices 8 at the other locations. The identification information of the object data and operator image 7 is not transmitted to the source electronic whiteboard 2. It may be transmitted to the terminal device 10 at the local location if this terminal device 10 is participating in the same conference.

[0186] S12: The communication unit 71 of the terminal device 8 at another location receives the object data and identification information of the operator image 7, and the display control unit 73 displays the object data on the display.

[0187] S13: In addition, when a user of a terminal device 8 at another location selects one of the displayed objects, the operator image acquisition unit 74 uses the identification information of the operator image 7 to request the operator image 7 from the information processing system 50 via the communication unit 71. The communication unit 51 of the information processing system 50 receives the request, and the object sharing unit 57 transmits the operator image 7, which is associated with the identification information of the operator image 7, to the terminal device 8 at the other location via the communication unit 51.

[0188] S14: The communication unit 71 of the terminal device 8 at another location receives the operator image 7, and the display control unit 73 displays the operator image 7 along with the identification information of the transmitted object. Displaying in association with an object means displaying it near the object (within a certain distance) or around it (above, below, left, and right).

[0189] With the above processing, terminal devices 8 at other locations will be able to see who wrote the strokes on the electronic whiteboard 2.

[0190] In the above example, the electronic whiteboard 2 transmits the operator image 7 to the information processing system 50 upon completion of the operation. However, the electronic whiteboard 2 may also transmit the operator image 7 to the information processing system 50 during the operation (while handwriting is in progress). Furthermore, although the object data and the operator image 7 are transmitted to the information processing system 50 almost simultaneously, they may be transmitted separately. For example, the electronic whiteboard 2 could first transmit the stroke data to the information processing system 50, and then save the operator image 7 to the information processing system 50. In other words, the stroke data could be shared first by terminal devices 8 at other locations, and then the operator image 7 captured when the stroke was input could be shared.

[0191] <Example of display on terminal devices at other locations> Figure 18 shows an example of an object and operator image 7 displayed by a terminal device 8 at another location. Figure 18(a) shows operator 121 writing on the electronic whiteboard 2. An operator image 7 is created showing operator 121 near the stroke 120 of the character "あ" written on the electronic whiteboard 2. The operator image 7 is the right half of the electronic whiteboard 2 because it was written on the right side of the electronic whiteboard 2, but the entire electronic whiteboard 2 could be cropped, or only the face could be cropped.

[0192] Figure 18(b) shows the stroke 120 and operator image 7 displayed on a terminal device 8 at another location. The operator image 7 is displayed as a speech bubble or pop-up near the handwritten "あ" on the electronic whiteboard 2. Therefore, users at other locations can easily understand who wrote the stroke 120.

[0193] In Figure 18(b), the meeting device 60 has deliberately cropped a wider area to make it easier for other locations to understand how the user is writing on the electronic whiteboard 2, or the overall atmosphere including the electronic whiteboard 2. As a result, the user's hands and a portion of the electronic whiteboard 2 are visible. Note that the display position of the operator image 7 is just an example; the operator image 7 only needs to be displayed around the stroke 120. The display control unit 73 may display the operator image 7 while avoiding existing strokes. Alternatively, the display control unit 73 may reduce the size of the operator image 7 to avoid existing strokes.

[0194] The meeting device 60 may use known image recognition (face recognition) technology to crop only the face or the area around the face into a circular or rectangular shape to generate the operator image 7. In Figure 18(b), the meeting device 60 obtains information from the electronic whiteboard 2 that something was handwritten on the right side of the electronic whiteboard 2, and generates the right half of the electronic whiteboard 2 as the operator image. Alternatively, the meeting device 60 may detect a person and generate an operator image that includes the electronic whiteboard 2 and the person.

[0195] Furthermore, terminal devices 8 at other locations can make the operator image 7 disappear after a certain period of time (e.g., when the electronic whiteboard 2 detects writing or when it determines that there has been no writing for several seconds or more), so that the operator image 7 does not become a hindrance. The display control unit 73 may also erase the image at the user's discretion within that period of time. Details of the display control of the operator image 7 are shown in Figure 38.

[0196] <An example of the process involved in communication between an electronic whiteboard and a terminal device> The following describes the process for communication between the electronic whiteboard and the terminal device. A user registers a new meeting with the information processing system 50 using a meeting registration screen 250, as shown in Figure 19. Figure 19 is an example of the meeting registration screen 250 displayed by the terminal device 10.

[0197] Item 241, "Meeting Name," is the title of the meeting.

[0198] Item 242, "Organizer," contains information about the person who organized the meeting, and is usually the person who operates the meeting registration screen.

[0199] The date and time field 243 is for entering the start and end dates and times of the meeting. The date and time field 243 can be selected from a calendar.

[0200] Participant item 244 is a participant in the meeting and can be selected from a list of users registered in the account information storage unit 5004.

[0201] Item 245 of Electronic Whiteboard 2 is a selection field for Electronic Whiteboard 2 to be used in the meeting, and in the account information storage unit 5004, it is possible to select from the list of accounts of Electronic Whiteboard 2, where the type is Electronic Whiteboard 2.

[0202] Item 246 for Meeting Device 60 is a selection field for the meeting device 60 to be used in the conference, and can be selected from a list of types designated as Meeting Device 60 in the account information storage unit 5004.

[0203] As shown in Figure 19, when creating a new meeting, the user can select the electronic whiteboard 2 and meeting device 60 to be used in that meeting.

[0204] Next, as shown in Figure 20, the user registers the meeting device 60 and the electronic whiteboard 2 with the information processing system 50. Since the meeting has already been created, the user only needs to select the meeting. Figure 20 is an example of a sequence diagram illustrating the process by which the user registers the meeting device 60 and the electronic whiteboard 2 with the information processing system 50.

[0205] S21: As shown in Figure 19, the user creates the meeting information.

[0206] S22: The operation reception unit 12 of the terminal device 10 receives the conference information, and the communication unit 11 transmits it to the information processing system 50. The communication unit 51 of the information processing system 50 receives the conference information, and the communication management unit 54 registers it in the conference information storage unit 5001.

[0207] S23: When a user starts a meeting, they may move to the meeting room and connect the meeting device 60 to the terminal device 10. The meeting device 60 starts up when the USB cable is connected, and communication with the meeting device 60 (transmission of various images) begins when the information recording application 41 is launched.

[0208] S24: The user selects the meeting to be held from the meeting list screen 230 (see Figure 33) displayed by the information recording app 41.

[0209] S25: The operation reception unit 12 of the information recording application 41 receives the operation, and the communication unit 11 requests registration to the meeting by specifying the equipment identification information and conference ID of the meeting device 60. The communication unit 51 of the information processing system 50 receives the request to register to the meeting and determines that it is a meeting device 60 that is registered in the meeting information. The equipment management unit 55 associates the equipment identification information of the meeting device 60 with the conference ID.

[0210] S26: Next, the user selects the meeting to be held from the meeting list screen 230 displayed on the electronic whiteboard 2.

[0211] S27: The communication unit 36 ​​of the electronic whiteboard 2 requests registration for the meeting, specifying the device identification information and meeting ID of the electronic whiteboard 2. The communication unit 51 of the information processing system 50 receives the registration request for the meeting and determines that it is the electronic whiteboard 2 registered in the meeting information. The device management unit 55 associates the device identification information of the electronic whiteboard 2 with the meeting ID.

[0212] In this way, the information processing system 50 associates the meeting device 60 (terminal device 10) with the electronic whiteboard 2, thereby enabling the terminal device 10 and the electronic whiteboard 2 to communicate with each other via the information processing system 50. When the meeting device 60 (terminal device 10) and the electronic whiteboard 2 transmit their respective identification information to the information processing system 50, the information processing system 50 can identify the other device registered with the same conference ID using the association information.

[0213] In Figures 19 and 20, the user has registered meeting information in advance, but even without registering meeting information beforehand, the user can associate the meeting device 60 and the electronic whiteboard 2 and register them with the information processing system 50 at the start of the meeting.

[0214] Figure 21 is an example of a sequence diagram illustrating the process by which a user registers a meeting device 60 and an electronic whiteboard 2 with the information processing system 50, even though no meeting information has been created yet.

[0215] S31: The user inputs an operation to start registering the association with the meeting device 60 on the electronic whiteboard 2 in the conference room.

[0216] S32: The electronic whiteboard 2 communicates with the pre-configured information processing system 50 in response to the operation, and registers the device identification information and the fact that it can be associated with the meeting.

[0217] S33: Furthermore, the code generation unit 35 of the electronic whiteboard 2, which is placed in the conference room used for meetings, generates a two-dimensional code encoded with its own device identification information and a statement indicating that it is device identification information, and the display control unit 34 displays it. The electronic whiteboard 2 may also include a password for authenticating other devices.

[0218] S34: The user enters the conference room where the electronic whiteboard 2 is installed, bringing the terminal device 10 and the meeting device 60, and connects the terminal device 10 and the meeting device 60 with a USB cable. The meeting device 60 starts up due to power supply from the USB cable or when the power is turned on. This puts the meeting device 60 into standby mode. The user also starts the information recording application 41 on the terminal device 10. When the information recording application 41 starts communicating with the meeting device 60, the meeting device 60 starts capturing images and collecting sound. The panoramic image generation unit 62 of the meeting device 60 generates image data of the surroundings, including a two-dimensional code.

[0219] S35: The terminal communication unit 61 of the meeting device 60 transmits image data and device identification information of the meeting device 60 to the terminal device 10.

[0220] S36: The device communication unit 16 of the terminal device 10 receives image data and device identification information of the meeting device 60, and the analysis unit 23 detects the two-dimensional code displayed by the electronic whiteboard 2 from the image data. The analysis unit 23 decodes the two-dimensional code and, if it determines that it contains information indicating that the device is usable for meetings, obtains the device identification information of the electronic whiteboard 2 from the two-dimensional code. The communication unit 11 of the information recording application 41 sends a registration request for the meeting to the information processing system 50, specifying the device identification information of the electronic whiteboard 2 and the device identification information of the meeting device 60. Note that decoding of the two-dimensional code is also possible on the meeting device 60.

[0221] S37, S38: When the communication unit 51 of the information processing system 50 receives a registration request for a meeting (device identification information, device identification information of the meeting device 60), the communication management unit 54 issues a meeting ID. The device management unit 55 then associates the device identification information of the electronic whiteboard 2, the device identification information of the meeting device 60, and the meeting ID, and stores them in the association information storage unit 5003.

[0222] The communication unit 51 of the information processing system 50 notifies the terminal device 10 and the electronic whiteboard 2 of the conference ID and the completion of registration for the conference on the electronic whiteboard 2. The communication unit 11 of the terminal device 10 receives and stores the conference ID. Similarly, when the communication unit 36 ​​of the electronic whiteboard 2 receives the conference ID and password, it verifies the password and stores the conference ID if they match. The terminal device 10 receives the conference ID as a response to the communication in step S36. The electronic whiteboard 2 receives the conference ID and password as a response to polling (requesting the mapping of device identification information to the conference) to the information processing system 50. The electronic whiteboard 2 and the information processing system 50 may communicate using a bidirectional communication method such as WebSocket, which enables push-type communication from the information processing system 50 to the electronic whiteboard 2.

[0223] From this point forward, terminal device 10 and electronic whiteboard 2 will attach the conference ID (or identification information for each device) to the data they transmit.

[0224] In this way, the information processing system 50 associates the meeting device 60 (terminal device 10) with the electronic whiteboard 2, thereby enabling the terminal device 10 and the electronic whiteboard 2 to communicate with each other via the information processing system 50.

[0225] In Figures 19 to 21, the terminal device 10 and the electronic whiteboard 2 can communicate via the information processing system 50. However, if the terminal device 10 and the electronic whiteboard 2 are connected to the same network, they can communicate using their IP addresses. The same network refers to cases where the network address is the same or the destination SSID (Service Set Identifier) ​​is the same.

[0226] Figure 22 is an example of a sequence diagram illustrating the communication method between terminal device 10 and electronic whiteboard 2 when terminal device 10 and electronic whiteboard 2 are connected to the same network.

[0227] S41: The user inputs an operation to start associating with the meeting device 60 on the electronic whiteboard 2 in the conference room.

[0228] S42: The electronic whiteboard 2 accepts the operation, and the display control unit 34 displays the IP address and password of the electronic whiteboard 2.

[0229] S43: The user sees the displayed IP address and password and enters them into the information recording application 41 of the terminal device 10. The operation reception unit 12 of the terminal device 10 accepts the input.

[0230] S44: The communication unit 11 of the terminal device 10 connects to the electronic whiteboard 2 using an IP address and sends a password. Upon successful password verification, the terminal device 10 and the electronic whiteboard 2 become able to communicate.

[0231] <Determining the orientation of the electronic whiteboard in a panoramic image> Next, we will explain how to determine the orientation of the electronic whiteboard 2 in the panoramic image. Since the operator image 7 is an image of the participant (user) operating the electronic whiteboard 2, if the meeting device 60 crops the image to show the orientation of the electronic whiteboard 2, an operator image 7 including the operator will be obtained.

[0232] There are mainly four methods for making a decision. 1. The user selects a panoramic image at the start of the meeting. 2. The electronic whiteboard 2 displays a specific image (such as a QR code), and the terminal device 10 or the meeting device 60 recognizes it from the panoramic image captured by the imaging unit 601 of the meeting device 60. 3. The electronic whiteboard 2 outputs a specific sound, which the meeting device 60 recognizes with the microphone 608. 4. The shape of the electronic whiteboard 2 is learned by an arbitrary information processing device using machine learning, and the terminal device 10 or the meeting device 60 recognizes it from the panoramic image captured by the camera (imaging unit 601) of the meeting device 60.

[0233] <<1. The user selects a panoramic image at the start of the meeting.>> Figure 23 illustrates the operation method for setting the orientation of the electronic whiteboard 2 by pressing the position registration button 207. When the position registration button 207 is pressed, a panoramic image 203 is displayed in a pop-up window. For example, the user moves a rectangular window 206 on the panoramic image 203 using a pointing device such as a mouse or touch panel. The user aligns the window 206 with the electronic whiteboard 2 or podium included in the panoramic image 203.

[0234] Figure 24 shows a screen for confirming the direction set by the user. When the user presses the OK button 208, the direction of the electronic whiteboard 2 relative to the panoramic image is set. The direction set by the user is transmitted to the meeting device 60, and the device direction detection unit 67 of the meeting device 60 stores it.

[0235] <<2. The interactive whiteboard displays a specific image (such as a QR code), and the terminal device or meeting device recognizes it from the panoramic image captured by the meeting device's imaging unit. 3. The interactive whiteboard outputs a specific sound, which the meeting device recognizes via its microphone.>> Figure 25 is an example of a sequence diagram illustrating the process by which the information recording application 41 determines the orientation of the electronic whiteboard 2 based on specific video or audio.

[0236] S51: The user inputs an operation to determine the direction of the electronic whiteboard 2 into the information recording application 41 of the terminal device 10. The operation reception unit 12 receives the operation.

[0237] S52: The communication unit 11 of the information recording application 41 requests the output of a specific video or a specific sound from the electronic whiteboard 2. The communication unit 11 may communicate with the electronic whiteboard 2 via the LAN or via the information processing system 50.

[0238] S53: The communication unit 36 ​​of the electronic whiteboard 2 receives a request, and the code generation unit 35 generates a two-dimensional code as a specific image. The display control unit 34 displays the two-dimensional code on the display 480.

[0239] Furthermore, the communication unit 36 ​​of the electronic whiteboard 2 receives a request, and the voice data generation unit 37 outputs a sound of a specific frequency from the speaker 450. Either the code generation unit 35 or the voice data generation unit 37 needs to be operational, but both may be operational.

[0240] S54: The panoramic image generation unit 62 of the meeting device 60 repeatedly generates panoramic images, so if a two-dimensional code is within the field of view, it automatically generates a panoramic image including the two-dimensional code. Also, the sound collection unit 64 of the meeting device 60 repeatedly collects sound, so it automatically collects sound of a specific frequency. The device direction detection unit 67 obtains a frequency spectrum by performing a Fourier transform on the audio data and identifies two directions from which sound of a predetermined frequency and above a threshold is arriving. The device direction detection unit 67 of the electronic whiteboard 2 stores the direction of the electronic whiteboard 2 (A° to B° within the 360° horizontal direction). It is preferable that this sound is in the ultrasonic frequency band, as it will not be audible to the user.

[0241] S55: The terminal communication unit 61 of the meeting device 60 transmits the panoramic image to the terminal device 10.

[0242] S56: The device communication unit 16 of the information recording application 41 receives the panoramic image. The analysis unit 23 detects the two-dimensional code displayed by the electronic whiteboard 2 within the panoramic image. The analysis unit 23 decodes the two-dimensional code and, if it determines that it contains information indicating the direction of the electronic whiteboard 2, it identifies the positions of the left and right edges of the two-dimensional code in the panoramic image. Details are explained in Figure 26. This process can also be performed by the device direction detection unit 67 of the meeting device 60.

[0243] S57: The device communication unit 16 of the terminal device 10 transmits the direction of the electronic whiteboard 2 (A° to B° within the 360° horizontal direction) to the meeting device 60. The terminal communication unit 61 of the meeting device 60 receives the direction of the electronic whiteboard 2, and the device direction detection unit 67 stores it.

[0244] S58: Now that the orientation of the electronic whiteboard 2 has been determined, the communication unit 11 of the information recording application 41 requests the electronic whiteboard 2 to stop outputting a specific video or a specific sound.

[0245] S59: The communication unit 36 ​​of the electronic whiteboard 2 receives the request, and the display control unit 34 terminates the display of the two-dimensional code. The audio data generation unit 37 stops outputting sound of a specific frequency.

[0246] <<Direction determination using a 2D code>> Figure 26 shows an example of a two-dimensional code 301 displayed as a specific image by the electronic whiteboard 2. In Figure 26, the electronic whiteboard 2 is visible in a panoramic image, and the electronic whiteboard 2 displays two two-dimensional codes 301. The size (width) of the two-dimensional code 301 is set to the size obtained by dividing the width of the electronic whiteboard 2's display into three parts. Two two-dimensional codes 301 are displayed. One is displayed right-aligned on display 480, and the other is displayed left-aligned. Each two-dimensional code 301 contains information indicating whether it is displayed on the right or left. In this embodiment, the right and left are determined based on the user's position facing the electronic whiteboard 2.

[0247] Figure 27 illustrates the method for determining direction based on the two-dimensional codes 301. The analysis unit 23 detects the two two-dimensional codes 301, thereby identifying the horizontal position information of each two-dimensional code 301. The analysis unit 23 determines that the position 253, which is moved to the left by the same amount as the width of the two-dimensional code 301 from the left end 251 of the left two-dimensional code 301, is the left edge position of the electronic whiteboard 2. The analysis unit 23 determines that the position 254, which is moved to the right by the same amount as the width of the two-dimensional code 301 from the right end 252 of the right two-dimensional code 301, is the right edge position of the electronic whiteboard 2. The analysis unit 23 determines the distance from the left end position 253 to the right end position 254 as the direction of the electronic whiteboard 2.

[0248] <<Direction determination by sound>> Figure 28 illustrates how the direction of the electronic whiteboard 2 is determined when the electronic whiteboard 2 emits a specific sound. As shown in Figure 28(a), speakers 450 are installed at the left and right ends of the electronic whiteboard 2. Speakers 450 may also be built into the left and right ends.

[0249] Then, as shown in Figure 28(b), the audio data generation unit 37 outputs sound from each speaker 450. The microphone 608 of the meeting device 60 is directional and identifies the direction from which the sound emitted from each speaker 450 is coming relative to the meeting device 60. Once the direction relative to the meeting device 60 is identified, the direction in the panoramic image 203 is also identified.

[0250] Furthermore, even if there is directionality, it is difficult for the device direction detection unit 67 to determine the center of the speaker 450. Therefore, the device direction detection unit 67 determines the position of the speaker 450 within a certain range 261 in the left-right direction, centered on the direction with the highest sound intensity, based on the sound intensity of a specific frequency.

[0251] As shown in Figure 28(c), the device direction detection unit 67 determines the area from the left end 262 to the right end 263 of the two ranges 261 as the direction of the electronic whiteboard 2.

[0252] <<4. Any information processing device learns the shape of the electronic blackboard through machine learning, and the terminal device or the meeting device recognizes it from the panoramic image captured by the imaging unit of the meeting device.>> Figure 29 is a diagram for explaining the method of determining the direction of the electronic blackboard 2 based on the electronic blackboard 2 detected by image processing such as machine learning. First, the device recognition unit 22 detects the shape (outer rectangle) of the electronic blackboard 2 by machine learning from the panoramic image. When the device recognition unit 22 detects the electronic blackboard 2 from the panoramic image, it divides the width of the electronic blackboard 2 into three equal parts, and determines the direction of the electronic blackboard 2 from the position 271 which is moved by one-third of the width from the left end of the electronic blackboard 2 to the position 272 which is moved by one-third of the width from the right end. One-third is just an example.

[0253] Since the terminal device 10 transmits the direction of the electronic blackboard 2 to the meeting device 60, the device direction detection unit 67 saves the direction of the electronic blackboard 2. Note that the position detection by image processing may be performed by the meeting device 60.

[0254] <Screen transition> Subsequently, referring to FIGS. 30 to 33, several screens displayed by the terminal device 10 during a remote meeting will be described. FIG. 30 is the initial screen 200 after login displayed by the information recording application 41 operating on the terminal device 10. The user of the terminal device 10 connects the information recording application 41 to the information processing system 50. When the user inputs the authentication information and succeeds in logging in, the initial screen 200 in FIG. 30 is displayed.

[0255] The initial screen 200 has a fixed display button 201, a front view change button 202, a panoramic image 203, one or more speaker images 204a to 204c (hereinafter referred to as speaker image 204 when not distinguished), a recording start button 205, and a location registration button 207. If the meeting device 60 is already activated and capturing images of the surroundings when the user logs in, the panoramic image 203 and speaker images 204 created by the meeting device 60 are displayed on the initial screen 200. Therefore, the user can decide whether or not to start recording while viewing these images. If the meeting device 60 is not activated (not capturing images), the panoramic image 203 and speaker images 204 are not displayed.

[0256] The information recording application 41 may display speaker images 204 of all participants based on all faces detected from the panoramic image 203, or it may display only the speaker images 204 of the N people who have spoken most recently. Figure 30 shows an example where up to three speaker images 204 are displayed. Until a participant speaks, there may be no speaker images 204 (one is added each time a participant speaks), or three speaker images 204 of participants facing a predetermined direction may be displayed (they are swapped depending on what is said).

[0257] If no one is speaking, such as immediately after the meeting device 60 is activated, a speaker image 204 will be created in a predetermined direction within the horizontal 360° (e.g., 0°, 120°, 240°). If a fixed display setting (described later) is enabled, that setting will take precedence.

[0258] The fixed display button 201 is a button that allows the user to perform an operation to fix a certain area of ​​the panoramic image 203 as the speaker image 204 and zoom in on it.

[0259] The front view change button 202 is a button for the user to change the front view of the panoramic image 203 (since the panoramic image captures 360° horizontally, the right and left edges coincide). The user can slide the panoramic image 203 left or right with a pointing device to determine which participant is in the front view. The user's operation is sent to the meeting device 60, which changes the angle that is considered the front view out of the 360° horizontal direction to create a panoramic image and sends it to the terminal device 10.

[0260] The position registration button 207 is a button used by the user to set the position (direction) of devices such as the electronic whiteboard 2.

[0261] When the user presses the recording start button 205, the information recording application 41 displays the recording settings screen 210 shown in Figure 31.

[0262] Figure 31 shows an example of the recording settings screen 210 displayed by the information recording application 41. On the recording settings screen 210, the user can configure whether to record (include in recording) the panoramic image and speaker image created by the meeting device 60, as well as the desktop screen or running application screen of the terminal device 10. If the information recording application 41 does not record either the panoramic image and speaker image, or the desktop screen or running application screen, only the audio (audio output by the terminal device 10 + audio collected by the meeting device 60) will be recorded.

[0263] The camera toggle button 211 is a button that switches the recording of panoramic images and speaker images created by the meeting device 60 on and off. The camera toggle button 211 may also be configured to record panoramic images and speaker images individually.

[0264] The PC screen toggle button 212 is a button that switches the recording of the desktop screen of the terminal device 10 and the screen of an application running on the terminal device 10 on and off. When the PC screen toggle button 212 is on, the desktop screen is recorded.

[0265] If a user wants to record an app screen, they further select an app from the app selection field 213. The app selection field 213 displays the names of apps currently running on the terminal device 10 in a dropdown format. The user can select the app to record. This app name is obtained by the information recording app 41 from the OS. The information recording app 41 can display only apps that have a UI (screen) among the running apps. The remote conferencing app 42 may be included in the selected apps. Therefore, the information recording app 41 can also record the materials displayed by the remote conferencing app 42 and the participants at each location in video. In addition, the apps displayed in the dropdown are various apps running on the terminal device, such as presentation apps, word processing apps, spreadsheet apps, document creation and editing apps, cloud electronic whiteboard apps, and web browser apps. Therefore, the user can flexibly select the app screens to include in the combined video.

[0266] Furthermore, when recording on an app-by-app basis, users can select multiple apps. The information recording app 41 can record the screens of all selected apps.

[0267] If both the camera toggle button 211 and the PC screen toggle button 212 are turned off, the recording content confirmation window 214 will display "Only audio will be recorded." This audio consists of the audio output by the terminal device 10 (audio received by the remote conferencing application 42 from other locations 101) and the audio collected by the meeting device 60. In other words, if a remote conference is being held, the audio from the remote conferencing application 42 and the audio from the meeting device 60 will be saved regardless of image recording. However, the user may selectively stop saving the audio from the remote conferencing application 42 and the audio from the meeting device 60 in their settings.

[0268] Depending on the on / off combination of the camera toggle button 211 and the PC screen toggle button 212, the combined image video is recorded as follows. The combined image video is also displayed in real time in the recording content confirmation window 214.

[0269] If the camera toggle button 211 is on and the PC screen toggle button 212 is off, the panoramic image captured by the meeting device 60 and the speaker image will be displayed in the recording content confirmation window 214.

[0270] If the camera toggle button 211 is off and the PC screen toggle button 212 is on (screen is also selected), the desktop screen or the screen of the selected application will be displayed in the recording content confirmation window 214.

[0271] When the camera toggle button 211 is on and the PC screen toggle button 212 is on, the recording content confirmation window 214 displays the panoramic image and speaker image captured by the meeting device 60, as well as the desktop screen and the screen of the selected application, side by side.

[0272] Therefore, there are cases where the panoramic image, speaker image, and app screen are not recorded, or where none of the panoramic image, speaker image, and app screen are recorded at all. However, in this embodiment, for convenience, the images created by the information recording app 41 are referred to as a combined image video.

[0273] Furthermore, the recording settings screen 210 has a checkbox 209 along with the message, "Automatically transcribe the recording after uploading it." The recording settings screen 210 also has a "Start Recording Now" button 217. When the user checks checkbox 209, the recorded video is attached with text data of the spoken words during the remote meeting. In this case, after the recording is finished, the information recording application 41 uploads the audio to the information processing system 50 along with a request for conversion to text data. Also, when the user presses the "Start Recording Now" button 217, the recording screen 220 shown in Figure 32 is displayed.

[0274] FIG. 32 is an example of a recording screen 220 displayed by the information recording application 41 during recording. In the description of FIG. 32, the differences from FIG. 30 will be mainly described. The recording screen 220 displays the combined image video to be recorded in real time under the conditions set by the user on the recording setting screen 210. The recording screen 220 in FIG. 32 is a case where the camera toggle button 211 is on and the PC screen toggle button 212 is off, and displays the panoramic image 203 and the speaker image 204 (both are videos) created by the meeting device 60. The recording screen 220 displays a recording icon 225, a pause button 226, and a recording end button 227.

[0275] The pause button 226 is a button for stopping the recording, and after stopping, it also accepts resuming the recording. The recording end button 227 is a button for ending the recording. The recording ID does not change when using the pause button 226, but changes when using the recording end button 227. When pausing and then resuming the recording, the user can also reset the recording conditions set on the recording setting screen 210 again. In that case, the information recording application 41 may create a plurality of video files each time the recording is stopped (for example, when the recording end button 227 is pressed), or may combine the plurality of files so as to be continuous as one video (for example, when the pause button 226 is pressed). Also, when the information recording application 41 plays back the combined image video, the plurality of video files may be played back continuously as one video.

[0276] The recording screen 220 also has a button 221 for retrieving information from the calendar, a meeting name field 222, a time field 223, and a location field 224. The button 221 for retrieving information from the calendar is used by the user to retrieve meeting information from the meeting management system 9. When the button 221 is pressed, the information recording application 41 retrieves and displays a list of meetings that the user has permission to view from the information processing system 50. The user selects the remote meeting to be held from the meeting list. As a result, the meeting information is reflected in the meeting name field 222, the time field 223, and the location field 224. The meeting name field 222 displays the title of the meeting information, the time field 223 displays the start and end times, and the location field 224 displays the location. In addition, the meeting information and recording information in the meeting management system are associated with each other by the meeting ID.

[0277] Once the remote meeting ends and the user stops recording, a combined video with audio is created.

[0278] Figure 33 shows an example of the meeting list screen 230 displayed by the information recording application 41. The meeting list screen 230 is a list of meetings, and can display a list of recorded information from remote meetings. It also includes meetings held only within a specific meeting room, regardless of whether they were remote meetings. The meeting list screen 230 displays meeting information that the logged-in user has permission to view in the meeting information storage unit 5001. Information from video recordings stored in the information storage unit 1001 may be further integrated.

[0279] The meeting list screen 230 is displayed when the user selects the meeting list tab 231 on the initial screen 200 in Figure 30. The meeting list screen 230 displays a list 236 of the recorded information that this user has permission to view. The meeting creator (minutes creator) can set viewing permissions for participants. The meeting list may be a list of saved recorded information, or a list of meeting schedules or meeting data.

[0280] The meeting list screen 230 has the following items: checkbox 232, update date and time 233, title 234, and status 235.

[0281] Checkbox 232 accepts the selection of video files. Checkbox 232 is used when the user wants to delete multiple video files at once.

[0282] • The update time 233 indicates the start and end times of the combined image video recording. If it has been edited, the edit time can be used instead.

[0283] • The title is 234, which is the title of the meeting (agenda, etc.). It can be copied from the meeting information or set by the user.

[0284] Status 235 indicates whether the combined image / video has been uploaded to the information processing system 50. If it has not been uploaded, "Local PC" is displayed; if it has been uploaded, "Uploaded" is displayed. If it has not been uploaded, the upload button is displayed. If there are unuploaded combined image / videos, the information recording application 41 should automatically upload them when the user logs into the information processing system 50.

[0285] When the user selects a title or other item from the list of combined images and videos 236 using the pointing device, the information recording application 41 displays a recording playback screen, but this is omitted in this embodiment. On the recording playback screen, it is possible to play back the combined images and videos.

[0286] Ideally, users should be able to filter meetings by update date, title, keywords, etc. Furthermore, if the number of displayed meetings is large and it is difficult to find the relevant one, a search function should be available allowing users to input terms to narrow down the recorded information based on keywords found in meeting statements or titles. This search function would enable users to quickly find the desired recorded information, even when dealing with a large volume of records. Additionally, the meeting list screen 230 may allow users to sort meetings by update date or title.

[0287] <Saving combined images and videos> Next, we will explain the process of saving the combined image video, referring to Figure 34. Figure 34 is an example of a sequence diagram showing the procedure by which the information recording application 41 records a panoramic image, a speaker image, and the application screen.

[0288] S61: The user starts a remote conference by operating the remote conferencing application 42. Here, it is assumed that the remote conferencing applications 42 at the local site 102 and the other site 101 have started a remote conference. The remote conferencing application 42 at local site 102 transmits the image captured by the camera and the audio collected by the microphone of the terminal device 10 to the remote conferencing application 42 at the other site 101. The remote conferencing application 42 at the other site 101 displays the received image on its display and outputs the received audio from its speaker. Similarly, the remote conferencing application 42 at the other site 101 transmits the image captured by the camera and the audio collected by the microphone of the terminal device 10 to the remote conferencing application 42 at local site 102. The remote conferencing application 42 at local site 102 displays the received image on its display and outputs the received audio from its speaker. Each remote conferencing application 42 repeats this process to realize a remote conference.

[0289] S62: The user configures the recording settings on the recording settings screen 210 of the information recording application 41 shown in Figure 31. The operation reception unit 12 of the information recording application 41 accepts the settings. Here, it is assumed that both the camera toggle button 211 and the PC screen toggle button 212 are turned on.

[0290] If a user has already scheduled a remote meeting, they can display a list of remote meetings by pressing the information acquisition button 221 on the calendar in Figure 32 and select the remote meeting to which they want to associate a recorded video. Since the user is logged into the information processing system 50, the information processing system 50 identifies the remote meetings that the logged-in user has access to. The information processing system 50 sends the list of identified remote meetings to the terminal device 10, and the user selects a remote meeting that is currently in progress or will be held soon. This determines the information related to the remote meeting, such as the meeting ID.

[0291] Furthermore, users can create a meeting when creating a combined image video, even if they haven't scheduled a remote meeting in advance. The following describes how the information recording application 41 creates a meeting when creating a combined image video and obtains the meeting ID from the information processing system 50.

[0292] S63: The user instructs the information recording application 41 to start recording (by pressing the "Start Recording Now" button 216). The operation reception unit 12 of the information recording application 41 receives the instruction. The display control unit 13 displays the recording screen 220.

[0293] S64: Since remote conferencing is not selected (because the conferencing ID has not been determined), the communication unit 11 of the information recording application 41 sends a remote conferencing creation request to the information processing system 50.

[0294] S65: The communication unit 51 of the information processing system 50 receives a remote meeting creation request, the communication management unit 54 obtains a unique meeting ID assigned by the meeting management system 9, and the communication unit 51 transmits the meeting ID to the information recording application 41.

[0295] S66: The communication management unit 54 also sends the destination of the combined image video (URL of the storage service system 70) to the information recording application 41 via the communication unit 51.

[0296] S67: The communication unit 11 of the information recording application 41 receives the meeting ID and the location where the video file is saved, and the communication unit 11 transmits the meeting ID to the electronic whiteboard 2. The communication unit 11 may transmit via the information processing system 50 or directly.

[0297] S68: When the communication unit 11 of the information recording application 41 receives the conference ID and the location where the video file will be saved, the recording control unit 17 determines that the recording is ready and starts recording.

[0298] S69: The application screen acquisition unit 14 of the information recording application 41 requests the application screen selected by the user (more specifically, the application screen acquisition unit 14 acquires the application screen via the OS). In Figure 34, the application selected by the user is the remote conferencing application 42.

[0299] S70: The recording control unit 17 of the information recording application 41 notifies the meeting device 60 to start recording via the device communication unit 16. When notifying, the recording control unit 17 may also notify that the camera toggle button 211 is on (request for panoramic image and speaker image). Regardless of whether a request is made, the meeting device 60 sends the panoramic image and speaker image to the information recording application 41.

[0300] S71: When the terminal communication unit 61 of the meeting device 60 receives a recording start notification, it assigns a unique recording ID and returns the recording ID to the information recording application 41. The recording ID may be assigned by the information recording application 41 or obtained from the information processing system 50.

[0301] S72: The audio acquisition unit 15 of the information recording application 41 acquires the audio data output by the terminal device 10 (audio data received by the remote conferencing application 42).

[0302] S73: The device communication unit 16 transmits the voice data acquired by the voice acquisition unit 15 and the synthesis request to the meeting device 60.

[0303] S74: The terminal communication unit 61 of the meeting device 60 receives voice data and a synthesis request, and the voice synthesis unit 65 synthesizes the received voice data with the surrounding voice data collected by the sound collection unit 64. For example, the voice synthesis unit 65 adds the two voice data together. Since clear sound from the surroundings of the meeting device 60 is recorded, the accuracy of text transcription of voices, especially those around the meeting device 60 (on the conference room side), is improved.

[0304] This voice synthesis can also be performed on the terminal device 10. However, by distributing the recording function to the terminal device 10 and the voice processing to the meeting device 60, the load on both the terminal device 10 and the meeting device 60 can be reduced. Alternatively, the recording function may be distributed to the meeting device 60 and the voice processing to the terminal device 10.

[0305] S75: In addition, the panoramic image generation unit 62 of the meeting device 60 creates a panoramic image, and the speaker image generation unit 63 creates a speaker image.

[0306] S76: The device communication unit 16 of the information recording application 41 repeatedly acquires panoramic images and speaker images from the meeting device 60. The device communication unit 16 also repeatedly requests and acquires synthesized audio data from the meeting device 60. These acquisitions may be performed by the device communication unit 16 making requests to the meeting device 60. Alternatively, the meeting device 60 may automatically transmit the panoramic images and speaker images upon receiving a request to synthesize audio data. The meeting device 60 may automatically transmit the synthesized audio data to the information recording application 41 upon receiving a request to synthesize audio data.

[0307] S77: The recording control unit 17 of the information recording application 41 creates a combined image by arranging the application screen acquired from the remote conferencing application 42, a panoramic image, and a speaker image. The recording control unit 17 repeatedly creates combined images and creates a combined image video by specifying each combined image as a frame that makes up the video. The recording control unit 17 also saves the audio data received from the meeting device 60.

[0308] The information recording application 41 repeats steps S72 to S77 described above. During steps S72 to S77, the process shown in Figure 17 (generation and transmission of operator image 7) is performed as needed.

[0309] S78: When the remote meeting ends and recording is no longer needed, the user instructs the information recording application 41 to stop recording (for example, by pressing the recording stop button 227). The operation reception unit 12 of the information recording application 41 receives the instruction.

[0310] S79: The device communication unit 16 of the information recording application 41 notifies the meeting device 60 that recording has ended. The meeting device 60 continues to create panoramic images and speaker images and synthesize audio. However, the meeting device 60 may change the processing load, such as changing the resolution or fps, depending on whether recording is in progress.

[0311] S80: The recording control unit 17 of the information recording application 41 combines audio data with the combined image video to create a combined image video with audio.

[0312] S81: Furthermore, if the user checks the checkbox 215 associated with "Automatically transcribe recording after uploading" on the recording settings screen 210, the audio data processing unit 18 requests the information processing system 50 to convert the audio data into text data. Specifically, the audio data processing unit 18 sends a request to the information processing system 50 via the communication unit 11 to specify the URL of the save location, along with the conference ID and recording ID, to convert the audio data combined into the combined image video.

[0313] S82: The communication unit 51 of the information processing system 50 receives a request to convert audio data, and the text conversion unit 56 converts the audio data into text data using the speech recognition service system 80. The communication unit 51 saves the text data to the same storage location (URL of the storage service system 70) as the storage location of the combined image video. In the recording information storage unit 5002, the text data is associated with the combined image video by the conference ID and recording ID. The text data may also be managed by the communication management unit 54 of the information processing system 50 and stored in the storage unit 5000. Alternatively, the terminal device 10 may request speech recognition from the speech recognition service system 80 and save the text data obtained from the speech recognition service system 80 to the storage location. The speech recognition service system 80 returns the converted text data to the information processing system 50, but may also send it directly to the storage URL. The speech recognition service system 80 may select or switch between multiple services according to the settings information set by the user in the information processing system 50.

[0314] S83: In addition, the upload unit 20 of the information recording application 41 saves the combined image video to the combined image video storage location via the communication unit 11. In the recording information storage unit 5002, the combined image video is associated with the conference ID and the recording ID. Uploaded combined image video is recorded.

[0315] S84: The user inputs "meeting ended" to the electronic whiteboard 2. Alternatively, the user may input "meeting ended" to the terminal device 10, and the terminal device 10 may transmit "meeting ended" to the electronic whiteboard 2. In this case, "meeting ended" may be transmitted to the electronic whiteboard 2 via the information processing system 50.

[0316] S85: The communication unit 36 ​​of the electronic whiteboard 2 specifies the meeting ID and transmits the saved object data displayed (e.g., handwritten) during the meeting to the information processing system 50. The object data itself is shared by the information processing system 50 in real time. The communication unit 36 ​​may also transmit the device identification information of the electronic whiteboard 2 to the information processing system 50. In this case, the meeting ID is identified by the correspondence information.

[0317] S86: Information processing system 50 saves object data to the same storage location as combined images and videos based on the conference ID.

[0318] Since the save location is notified to the user, the user can share the combined image / video with participants by informing them of the save location via email or other means. Even if the devices used to create the combined image / video, audio data, text data, and object data are different, they can be collected and stored in a single location, making them easily viewable by users later.

[0319] Note that the processing in steps S72 to S77 does not have to be in the order shown in Figure 34, and the order in which the audio data is synthesized and the combined image is created may also be reversed.

[0320] <Main effects> In this embodiment, the operator image 7 is displayed near the stroke, making it easy for users at other locations to understand who is writing (or has written) it. Furthermore, in the information processing system 50, the operator image 7 is associated with the object data, making it clear who entered the object and improving the quality of the meeting minutes. [Examples]

[0321] This embodiment describes a case where information is sent and received via the information processing system 50, rather than directly communicating between the electronic whiteboard 2 and the terminal device 10. Even if the electronic whiteboard 2 and the terminal device 10 cannot communicate directly, a terminal device 8 at another location can display the operator image 7.

[0322] In this embodiment, it is assumed that the hardware configuration diagrams shown in Figures 4, 5, and 8, and the functional block diagram shown in Figure 9, as described in the above embodiment, can be used as reference.

[0323] Figure 35 is a sequence diagram illustrating the process by which a terminal device 8 at another location displays strokes handwritten by a user on the electronic whiteboard 2, along with the operator's image 7. Note that the explanation of Figure 35 primarily focuses on the differences from Figure 17.

[0324] S91, S92: Users participating in the meeting can write by hand using the electronic pen 490 or display their PC screen on the electronic whiteboard 2. The communication unit 36 ​​of the electronic whiteboard 2 transmits the start of operation to the information processing system 50 along with the device identification information of the electronic whiteboard 2, as the electronic whiteboard 2 has been operated.

[0325] S93: The communication unit 51 of the information processing system 50 transmits an operation start message to the meeting device 60 (i.e., terminal device 10) associated with the device identification information of the electronic whiteboard 2. The terminal device 10 repeatedly polls the information processing system 50 with the meeting ID and receives communication from the information processing system 50. Alternatively, the terminal device 10 may communicate with the information processing system 50 using bidirectional communication such as WebSockets.

[0326] The processing in S94-S96 can be the same as in S3-S5 in Figure 17.

[0327] S97: The user has finished writing.

[0328] S98: The operation detection unit 38 of the electronic whiteboard 2 detects the end of the operation (a request to save a cropped image of the direction of the electronic whiteboard). In response to the end of the operation, the operator image receiving unit 40b of the electronic whiteboard 2 requests the information processing system 50 for the operator image 7 via the communication unit 36, specifying the device identification information of the electronic whiteboard 2.

[0329] S99: The communication unit 51 of the information processing system 50 receives the device identification information of the electronic whiteboard 2 and a request for the operator image 7, and the object sharing unit 57 requests the operator image 7 from the terminal device 10 connected to the meeting device 60 associated with the electronic whiteboard 2 via the communication unit 51.

[0330] S100: The communication unit 11 of the terminal device 10 receives a request for the operator image 7, and the operator image transmission unit 24 transmits the operator image 7 to the information processing system 50 via the communication unit 11.

[0331] S101: The communication unit 51 of the information processing system 50 receives the operator image 7, and the object sharing unit 57 assigns an identification number to the operator image 7. The object sharing unit 57 stores the operator image 7 in association with its identification information. The object sharing unit 57 also transmits the identification information of the operator image 7 to the electronic whiteboard 2 that transmitted the operation completion message via the communication unit 51.

[0332] S102: The communication unit 36 ​​of the electronic whiteboard 2 has received the identification information of the operator image 7, so the object upload unit 40a transmits the conference ID, object data, and the identification information of the operator image 7 to the information processing system 50 via the communication unit 36.

[0333] S103: The communication unit 51 of the information processing system 50 receives the conference ID, object data, and identification information of the operator image 7, and the object sharing unit 57 stores the object data, the identification information of the operator image 7, and the operator image 7 in association. The object ID may be assigned by the electronic whiteboard 2 or by the information processing system 50.

[0334] S104: The object sharing unit 57 also transmits object data and identification information of the operator image 7 to terminal devices 8 at other locations participating in the same conference, identified by the conference ID, via the communication unit 51. The process thereafter is the same as in Figure 17.

[0335] <Main effects> According to this embodiment, in addition to the effects of Embodiment 1, even if the terminal device 10 and the electronic whiteboard 2 cannot communicate directly, the operator image 7 is transmitted from the terminal device 10 to the electronic whiteboard 2 via the information processing system 50. [Examples]

[0336] This embodiment describes a case where one user writes strokes consecutively, and then another user writes strokes. Assuming continuous processing as in this embodiment, the timing of image acquisition and other aspects differ from those in Embodiments 1 and 2.

[0337] In this embodiment, it is assumed that the hardware configuration diagrams shown in Figures 4, 5, and 8, and the functional block diagram shown in Figure 9, as described in the above embodiment, can be used as reference.

[0338] Figure 36 is a sequence diagram illustrating the process by which a terminal device 8 at another location displays the strokes along with the operator image 7 when user 1 writes on the electronic whiteboard 2, and then another user 2 writes on it. First, the process in steps S111 to S116 is the same as in steps S91 to S96 in Figure 35.

[0339] Furthermore, the processing in steps S117 to S120 can be the same as in S98 to S101 in Figure 35, but in Figure 36, it is assumed that strokes are processed continuously, so the identification information of the operator image 7 is acquired without waiting for the electronic whiteboard 2 to finish the operation.

[0340] S121: The communication unit 36 ​​of the electronic whiteboard 2 receives the identification information of the operator image 7 and stores it in the object data storage unit 3002. For the sake of explanation, the identification information of the operator image 7 is set to 1.

[0341] S122: The user finishes writing.

[0342] S123: The operation detection unit 38 of the electronic whiteboard 2 detects the end of the operation, and the object upload unit 40a transmits the identification information of the operator image 7 and the object data to the information processing system 50 via the communication unit 36.

[0343] S124: The communication unit 51 of the information processing system 50 receives the identification information of the operator image 7 and object data, the object sharing unit 57 assigns an object ID (which may be assigned by the electronic whiteboard 2), and stores it in the object data storage unit 5005 in association with the identification information of the operator image 7.

[0344] S125: The object sharing unit 57 also transmits object data and identification information of the operator image 7 to terminal devices 8 at other locations participating in the same conference via the communication unit 51. The process thereafter is the same as in Figure 17.

[0345] S126: Next, user 1 starts writing stroke 2. Since the input starts (pen down) within a certain time after the end of input (pen up) of stroke 1 (the last stroke), the operator image receiving unit 40b determines that the writing of stroke 2 is by the same user and does not acquire operator image 7. The operator image receiving unit 40b has ID=1, which was acquired during stroke 1, so it can use the same operator image 7.

[0346] S127~S130: User 1 finishes writing stroke 2. This process is the same as in steps S122~S125. The identification information for operator image 7 is ID=1.

[0347] S131: User 2 begins writing stroke 3. Since the input begins (pen down) after a certain amount of time has elapsed since the end of input (pen up) of stroke 2 (the last stroke), the operator image receiving unit 40b determines that user 2 is a different user and obtains the identification information of operator image 7. The operator image receiving unit 40b determines that the person writing may have changed to another user and therefore the image needs to be updated.

[0348] In addition to detecting a change in operator based on whether a certain amount of time has elapsed between pen-up and pen-down, the operator image receiving unit 40b may also detect a change in operator through image processing. Although the face of the user operating the electronic whiteboard 2 is not displayed on the meeting device 60, the operator image receiving unit 40b can detect a change in operator based on the characteristics of the user's back (height, hairstyle, clothing, etc.).

[0349] In steps S132 to S141, the same processing as in steps S112 to S121 is performed. However, the identification information of operator image 7 is newly assigned an ID=2.

[0350] S142~S145: User 2 finishes writing stroke 3. This process is the same as in steps S122~S125. The identification information for operator image 7 is ID=2.

[0351] Thus, according to this embodiment, the electronic whiteboard 2 can estimate that the user has changed and acquire identification information of the operator image 7 each time.

[0352] Figure 37 shows the operator image 7 displayed by the terminal device 8 at another location when two users 1 and 2 are operating the electronic whiteboard 2. Figure 37(a) shows the two users 1 and 2 operating the electronic whiteboard 2. Figure 37(b) shows the operator image 7 displayed by the terminal device 8 at another location. As explained in Figure 36, while user 1 is writing the stroke 120 for "あ", the terminal device 8 at the other location displays the operator image 7 only once and then erases it after a certain period of time.

[0353] Next, when user 2 begins to write the stroke 122 for "i", the terminal device 8 at the other location acquires and displays a new operator image 7. If the operator image 7 associated with "a" is currently being displayed, the display control unit 73 erases the currently displayed operator image 7 (Figure 36(b)) and displays the operator image 7 associated with "i" (Figure 36(c)).

[0354] <Processing at terminal devices in other locations> Next, referring to Figure 38, the process by which the terminal device 8 at another location displays the operator image 7 will be explained. Figure 38 is a flowchart illustrating the process by which the terminal device 8 at another location, upon receiving the operator image 7, displays the operator image 7. The following processes are performed by the terminal device 8 at the other location.

[0355] First, the display control unit 73 displays the stroke and operator image 7 received from the information processing system 50 (S201).

[0356] Next, the communication unit 71 determines whether or not it has received the next stroke data and the identification information of the operator image 7 (S202). The stroke data is accompanied by the identification information of the operator image 7, but as mentioned above, the identification information of the operator image 7 may be the same or different. If the determination in step S202 is No, the process proceeds to step S206.

[0357] If the determination in step S202 is Yes, the operator image acquisition unit 74 determines whether the identification information of the operator image 7 is the same (S203).

[0358] If the identification information of operator image 7 is the same as the operator image 7 currently being displayed (Yes in S203), there is no need to acquire operator image 7, so the process proceeds to step S205.

[0359] If the identification information of operator image 7 is not the same as the operator image 7 currently being displayed (No. in S203), the display control unit 73 switches operator image 7 (S204). That is, the operator image acquisition unit 74 acquires operator image 7 from the information processing system 50 by specifying the identification information of operator image 7 via the communication unit 71. Even if the identification information of operator image 7 is different, the operator image 7 associated with the newly added stroke may show the same operator as the operator image 7 before the switch, or it may show a different operator.

[0360] Furthermore, the display control unit 73 erases the operator image 7 (S206) after a certain period of time has elapsed since the display of the operator image 7 (Yes in S205).

[0361] Terminal devices 8 at other locations retain the operator image 7 in their object data even after it has been displayed and then dismissed, allowing it to be redisplayed. This makes it possible, for example, to redisplay the operator's image when a user touches a stroke.

[0362] Figure 39 shows a mark 130 indicating that an operator image 7 is attached to a stroke 120 displayed by a terminal device 8 at another location. As described above, the operator image 7 disappears after a certain period of time has elapsed since its display, but the user may want to display the operator image 7 again. If the user has a stroke 120 for which they want to display the operator image 7, they can display the mark 130 by pressing it with a pointing device 131 such as a mouse, pen, or finger. When the user presses the mark 130 with the pointing device 131, the display control unit displays the operator image 7 associated with the stroke 120. This operator image 7 is stored on the terminal device 8 at another location if a meeting is in progress. The user can check the operator of an object at any time.

[0363] Furthermore, the display control unit 73 may display the operator image 7 again simply by the user pressing stroke 120.

[0364] If, after the meeting has ended, a terminal device 8 at another location displays an object acquired from the information processing system 50, the operator image 7 may not be transmitted to the terminal device 8 at the other location along with the object data because of its large file size. In this case, the operator image acquisition unit 74 can acquire the operator image 7 from the information processing system 50 based on the identification information of the operator image 7.

[0365] Note that mark 130 is just an example, and the stroke 120 to which the operator image 7 is associated may blink or change color. Also, mark 130 may be displayed not by pressing, but by hovering the mouse over the pointing device 131.

[0366] <Input display area> Figure 40 is a diagram illustrating the input display area. The object sharing unit 57 shares the input display area, object data, and operator image 7 between the terminal device 8 and the electronic whiteboard 2, which are participating in the same meeting. As shown in Figures 40(a) and 40(b), the object sharing unit 57 allows the electronic whiteboard 2 and the terminal device 8 to share the input display area 270 and accepts object inputs to the input display area 270 from the electronic whiteboard 2 and the terminal device 8. The object sharing unit 57 transmits objects sent from one electronic whiteboard 2 to the other electronic whiteboard 2 in response to object input, and vice versa.

[0367] <Other application examples> Although the best mode for carrying out the present invention has been described above using examples, the present invention is not limited in any way to these examples, and various modifications and substitutions can be made without departing from the spirit of the present invention.

[0368] For example, the range of the operator's image in the panoramic image may be determined by detecting a person near the electronic whiteboard 2 using image processing. Alternatively, a meeting device or terminal device may use image processing to detect that a person near the electronic whiteboard 2 is writing on it.

[0369] Furthermore, the terminal device 10 and the meeting device 60 may be integrated into a single unit. Alternatively, the meeting device 60 may be attached externally to the terminal device 10. Also, the meeting device 60 may consist of a 360-degree camera, microphone, and speaker connected by cables.

[0370] Meeting devices 60 may also be placed at other locations 101. These other locations 101 will separately use meeting devices 60 to create combined image / video and text data. Furthermore, multiple meeting devices 60 may be placed at a single location. In this case, multiple recordings will be created for each meeting device 60.

[0371] Furthermore, the arrangement of the panoramic image 203, speaker image 204, and application screen in the combined image video used in this embodiment is merely an example. The panoramic image 203 may be at the bottom and the speaker image 204 at the top, or the user may change the arrangement, or individually switch the display of the panoramic image 203 and speaker image 204 during playback.

[0372] Furthermore, the configuration examples shown in Figure 9 and other figures are divided according to their main functions to facilitate understanding of the processing performed by the terminal device 10, meeting device 60, and information processing system 50. The present invention is not limited by the way the processing units are divided or the names of those units. The processing of the terminal device 10, meeting device 60, and information processing system 50 can be further divided into more processing units depending on the processing content. Alternatively, a single processing unit can be divided to include even more processing.

[0373] Furthermore, the apparatus described in the examples represents only one of several computing environments for carrying out the embodiments disclosed herein. In one embodiment, the information processing system 50 includes multiple computing devices, such as a server cluster. The multiple computing devices are configured to communicate with each other via any type of communication link, including networks and shared memory, and perform the processing disclosed herein.

[0374] Furthermore, the information processing system 50 can be configured to share the disclosed processing steps, such as those shown in Figure 17, in various combinations. For example, a process executed by a predetermined unit can be executed by multiple information processing devices of the information processing system 50. Also, the information processing system 50 may be consolidated into a single server device or divided into multiple devices.

[0375] Each of the embodiments described above can be implemented by one or more processing circuits. Here, "processing circuit" as used herein includes processors programmed to execute each function by software, such as processors implemented by electronic circuits, as well as devices such as ASICs (Application Specific Integrated Circuits), DSPs (Digital Signal Processors), FPGAs (Field Programmable Gate Arrays), and conventional circuit modules designed to execute each of the functions described above. [Explanation of symbols]

[0376] 10 Terminal devices 50 Information Processing Systems 60 Meeting Devices 100 Record Information Creation System [Prior art documents] [Patent Documents]

[0377] [Patent Document 1] Japanese Patent Publication No. 2016-122226

Claims

1. A camera unit for capturing images of a remote conference, A detection unit for detecting the actions of participants in the aforementioned remote conference, An image generation unit generates an image of a portion of the remote conference image, including an object and the participant, based on an image that determines the range included in the remote conference image according to the participant's actions, The system includes a display unit that displays an image of a portion of the aforementioned region. The image generation unit generates an image of a portion of the region based on the ranges indicated by a plurality of images that determine the ranges on the object. A device system characterized by the following features.

2. Multiple images determining the aforementioned ranges indicate the ranges of the right and left edges of the image of the partial region. The image generation unit generates an image of the partial region including a plurality of images that determine the range. The equipment system according to claim 1, characterized in that it is as described above.

3. The images that determine the multiple ranges are two-dimensional codes. The equipment system according to claim 1, characterized in that it is as described above.

4. The images that determine the multiple ranges are displayed on the object. The equipment system according to claim 1, characterized in that it is as described above.

5. The detection unit detects the participant's actions toward the object. The equipment system according to claim 4, characterized in that it is as described above.

6. The detection unit detects an instruction from a participant in the remote conference to terminate the generation of an image of a portion of the region based on a plurality of images that determine the range included in the image of the remote conference. The equipment system according to claim 1, characterized in that it is as described above.

7. A device connected to a terminal device equipped with a display unit, The photography department, which takes images of the remote meeting, A detection unit for detecting the actions of participants in the aforementioned remote conference, An image generation unit generates an image of a portion of the remote conference image, including an object and the participant, based on an image that determines the range included in the remote conference image according to the participant's actions, The system includes a transfer unit that transfers images of a portion of the aforementioned region to the terminal device. The image generation unit generates an image of a portion of the region based on the ranges indicated by a plurality of images that determine the ranges on the object. A device characterized by the following features.

8. Multiple images determining the aforementioned ranges indicate the ranges of the right and left edges of the image of the partial region. The image generation unit generates an image of the partial region including a plurality of images that determine the range. The device according to feature 7.

9. The images that determine the multiple ranges are two-dimensional codes. The device according to feature 7.

10. The images that determine the multiple ranges are displayed on the object. The device according to feature 7.

11. The detection unit detects the participant's actions toward the object. The device according to feature 10.

12. The detection unit detects an instruction from a participant in the remote conference to terminate the generation of an image of a portion of the region based on a plurality of images that determine the range included in the image of the remote conference. The device according to feature 7.

13. A device connected to a terminal device equipped with a display unit, The photography department, which takes images of the remote meeting, A detection unit for detecting the actions of participants in the aforementioned remote conference, An image generation unit generates an image containing objects and the participants by cutting out the image of the remote conference based on an image that determines the range included in the image of the remote conference according to the actions of the participants, The system includes a transfer unit that transfers the cropped image to the terminal device. The image generation unit generates the cropped image based on the range indicated by the multiple images that determine the range on the object. A device characterized by the following features.

14. Multiple images determining the aforementioned ranges indicate the ranges of the rightmost and leftmost edges of the cropped image. The image generation unit generates the cropped image which includes a plurality of images that determine the range. The device according to claim 13, characterized in that it is a device characterized by the features described above.

15. The images that determine the multiple ranges are two-dimensional codes. The device according to feature 13.

16. The images that determine the multiple ranges are displayed on the object. The device according to feature 13.

17. The detection unit detects the participant's actions toward the object. The device according to Feature 16.

18. The detection unit detects an instruction from a participant in the remote conference to terminate the generation of the cropped image based on multiple images that determine the ranges included in the image of the remote conference. The device according to feature 13.

19. A terminal device equipped with a display unit, The acquisition unit acquires images of the remote meeting taken by the photography unit, An image generation unit generates an image containing objects and participants by cutting out an image of the remote conference based on an image that determines the range included in the image of the remote conference according to the actions of the participants in the remote conference, The system includes a control unit that displays the cropped image on the display unit. The image generation unit generates the cropped image based on the range indicated by the multiple images that determine the range on the object. A terminal device characterized by the following features.

20. A device connected to a terminal device equipped with a display unit, and an image generation method performed by said device, The photography department handles the process of capturing images from the remote meeting, The detection unit processes the actions of the participants, The image generation unit performs a process in which, in accordance with the actions of the participants, it generates an image of a portion of the remote conference image, including objects and the participants, based on an image that determines the range included in the remote conference image. The transfer unit performs the process of transferring the image of the part of the region to the terminal device, The image generation unit performs the process of generating an image of a portion of the region based on the range indicated by a plurality of images that determine the range on the object, An image generation method characterized by performing the following.

21. A device connected to a terminal device equipped with a display unit, The photography department, which takes images of the remote meeting, A detection unit for detecting the actions of participants in the aforementioned remote conference, An image generation unit generates an image of a portion of the remote conference image, including an object and the participant, based on an image that determines the range included in the remote conference image according to the participant's actions, It functions as a transfer unit that transfers images of the aforementioned partial region to the terminal device, The image generation unit generates an image of a portion of the region based on the ranges indicated by a plurality of images that determine the ranges on the object. A program characterized by the following features.

22. The device according to claim 7, characterized in that the detection unit detects the actions of the participants based on the images of the remote conference.

23. The device according to claim 7, characterized in that the image of the part of the region is a video.

24. The device according to claim 7, characterized in that the image of the part of the region includes a region outside the image that determines the range.

Images