Display system, display method, imaging device, program

Abstract

To provide technique of displaying display device image information of a specific area including at least a portion of a display device from image information captured by an imaging apparatus.SOLUTION: A display system comprises: a display device identification unit which identifies a display device based on information on the display device; a detection unit which detects that a user is using the display device; and a display control unit which displays a display device image of a specific area including at least a portion of the display device from image information captured by an imaging apparatus.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present invention relates to a display system, a display method, an imaging device, and a program.

Background Art

[0002] A remote communication system that transmits images and sounds in real time from one base to one or more other bases and enables users at remote locations to hold meetings using images and sounds is known. In remote communication, devices such as an electronic blackboard may be used.

[0003] Techniques for recording that the topic of a meeting has changed have been devised (see, for example, Patent Document 1). Patent Document 1 discloses a technique for associating and holding meaning information such as that the topic of a meeting has changed at a specific timing during a meeting with the elapsed time.

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in the conventional technology, when an imaging device that images the surroundings of an imaging device and a display device such as an electronic blackboard are arranged in a meeting room, although participants (speakers) and the display device may be shown in the image captured by the imaging device, it was not possible to transmit to a remote location the state of the display device and its vicinity when a user is using the display device.

[0005] In view of the above problems, an object of the present invention is to provide a technique for displaying a display device image of a specific area including at least a part of a display device from image information captured by an imaging device.

Means for Solving the Problems

[0006] In view of the above problems, the present invention A display system having a display device 、 The aforementionedA display device identification unit identifies the display device based on information about the display device; a detection unit detects that a user is using the display device; and a display control unit displays a display device image of a specific region including at least a part of the display device from image information captured by an imaging device. The display device has a communication unit that transmits to a terminal device that can communicate with the imaging device that the detection unit has detected that the user is using the display device, the detection unit detects the contact position of the display device with respect to the touch panel, the communication unit transmits the contact position detected by the detection unit to the terminal device, and the display control unit displays a display device image that has been modified according to the contact position received from the display device, wherein if the user has written to the right half of the touch panel, the display device image includes the right half; if the user has written to the left half of the touch panel, the display device image includes the left half; and if the user has written to both the left and right areas of the touch panel, the display device image includes the entire display device. [Effects of the Invention]

[0007] This technology provides a method for displaying a display device image of a specific region, including at least a portion of the display device, from image information captured by an imaging device. [Brief explanation of the drawing]

[0008] [Figure 1] This diagram outlines the creation of recording information that saves the screen of an application running during a remote meeting, along with a panoramic image of the surroundings. [Figure 2] This diagram illustrates the process by which a meeting device extracts an image from a display device showing an electronic whiteboard. [Figure 3] This diagram shows an example configuration of a record information creation system. [Figure 4] This figure shows an example of the hardware configuration of an information processing system and terminal device. [Figure 5] This figure shows an example of a hardware configuration for a meeting device. [Figure 6] This diagram illustrates the imaging range of the meeting device. [Figure 7] This diagram illustrates the process of extracting images from a panoramic image and a speaker image. [Figure 8] This is a hardware configuration diagram of an example of an interactive whiteboard. [Figure 9] This is an example of a functional block diagram that explains the functions of terminal devices, meeting devices, and information processing systems in a record information creation system by dividing them into blocks. [Figure 10] This figure shows an example of video recording information stored in the information storage unit. [Figure 11]It is a diagram showing an example of meeting information managed by the communication management department. [Figure 12] It is a diagram showing an example of association information in which a meeting ID and device identification information are associated and stored in the association information storage unit. [Figure 13] It is a diagram showing an example of account information stored in the account information storage unit. [Figure 14] It is an example of a functional block diagram that explains the functions of an electronic blackboard divided into blocks. [Figure 15] It is a diagram showing an example of device identification information and the like stored in the device information storage unit. [Figure 16] It is a diagram explaining the object information stored in the object information storage unit. [Figure 17] It is a diagram showing an example of the initial screen after login displayed by the information recording app operating on the terminal device. [Figure 18] It is a diagram showing an example of the recording setting screen displayed by the information recording app. [Figure 19] It is a diagram showing an example of the screen during recording displayed by the information recording app during recording. [Figure 20] It is a diagram showing an example of the meeting list screen displayed by the information recording app. [Figure 21] It is an example of a sequence diagram explaining the process in which the meeting device starts and ends the extraction of the display device image in response to the start and end of the operation of the electronic blackboard. [Figure 22] It is a diagram showing an example of the meeting registration screen displayed by the terminal device. [Figure 23] It is an example of a sequence diagram explaining the process in which a user registers a meeting device and an electronic blackboard in the information processing system. [Figure 24] It is an example of a sequence diagram explaining the process in which a user registers a meeting device and an electronic blackboard in the information processing system when no meeting information has been created. [Figure 25]This is an example of a sequence diagram explaining the communication method between a terminal device and an electronic blackboard when they are connected to the same network. [Figure 26] This is a diagram explaining the operation method of setting the direction of the electronic blackboard by pressing the location registration button. [Figure 27] This is a diagram showing the screen for confirming the direction set by the user. [Figure 28] This is an example of a sequence diagram explaining the process in which an information recording application determines the direction of the electronic blackboard based on specific video or audio. [Figure 29] This is a diagram showing an example of a two-dimensional code that the electronic blackboard displays as a specific video. [Figure 30] This is a diagram explaining the method of determining the direction based on the two-dimensional code. [Figure 31] This is a diagram explaining the method of determining the direction of the electronic blackboard when the electronic blackboard outputs a specific sound. [Figure 32] This is a diagram explaining the direction of determining the direction of the electronic blackboard based on the electronic blackboard detected by image processing such as machine learning. [Figure 33] This is an example of a state transition diagram based on menu operations and writing on the electronic blackboard. [Figure 34] This is an example of a state transition diagram that is the same in terms of menu operations and writing, but also determines on which side (left or right) of the display the menu operations and writing are located. [Figure 35] This is an example of a flowchart diagram explaining the process in which the electronic blackboard determines the transition to the operation start state. [Figure 36] This is an example of a flowchart diagram explaining the process in which the recording information creation system determines the transition to the operation end state. [Figure 37] This is an example of a state transition diagram based on the user's approach to the electronic blackboard. [Figure 38] This is an example of a flowchart diagram explaining the process in which the recording information creation system determines the transition to the operation start state. [Figure 39]This is an example of a flowchart illustrating the process by which a record information creation system determines the transition to the operation completion state. [Figure 40] This figure shows an example of a predetermined orientation of the electronic whiteboard in a panoramic image. [Figure 41] This figure shows an example of a display image cropped based on the orientation of the electronic whiteboard. [Figure 42] This figure shows an example of a half-display image when a request to start recording an electronic whiteboard, specifying either the left or right side of the electronic whiteboard, is sent to a meeting device. [Figure 43] This figure shows an example of how the speaker image is displayed by the display control unit of the information recording application. [Figure 44] The image on the right is an example of how the display control unit of the information recording application displays a half-display image when a specified electronic whiteboard recording start request is sent to the meeting device. [Figure 45] This is an example of how the display control unit of the information recording application displays the display image when an electronic whiteboard recording start request without left / right orientation is sent to the meeting device. [Figure 46] This is an example of a diagram showing a display device image that is prioritized and displayed based on user input. [Modes for carrying out the invention]

[0009] The following describes a display system and a display method performed by the display system as an example of an embodiment for carrying out the present invention.

[0010] <An example of how to create meeting minutes for remote meetings> First, with reference to Figure 1, we will outline the method for creating meeting minutes using panoramic images and application screens. Figure 1 is a diagram illustrating the outline of creating recording information that saves application screens executed during a remote meeting along with panoramic images of the surroundings. As shown in Figure 1, a user at the local site 102 is conducting a remote meeting with another site 101 using the remote conferencing service system 90.

[0011] The recording information creation system 100 of this embodiment creates recording information (such as meeting minutes) using a horizontal panoramic image (hereinafter referred to as the panoramic image) acquired by processing information captured by a meeting device 60 equipped with an imaging means capable of capturing images of the surroundings in a 360° range, a microphone, and a speaker, and a screen created by an application (hereinafter referred to as the app) executed by the terminal device 10. For audio, the recording information creation system 100 combines the audio received by the remote conferencing app 42 and the audio acquired by the meeting device 60 and includes them in the recording information. The outline is described below.

[0012] (1) The terminal device 10 is running the information recording application 41 and the remote conferencing application, which will be described later. Other applications, such as one for displaying documents, may also be running. The information recording application 41 transmits the audio output by the terminal device 10 (including audio received by the remote conferencing application from other locations; an example of the first audio data) to the meeting device 60. The meeting device 60 mixes (combines) the audio it has acquired (an example of the second audio data) with the audio from the remote conferencing application.

[0013] (2) The meeting device 60 is equipped with a microphone and, based on the direction from which the sound was acquired, processes the image to extract the speaker from the panoramic image and creates a speaker image. The meeting device 60 transmits both the panoramic image and the speaker image to the terminal device 10.

[0014] (3) The information recording application 41 running on the terminal device 10 can display the panoramic image 203 and the speaker image 204. The information recording application 41 combines the panoramic image 203 and the speaker image 204 with any application screen selected by the user (for example, the remote conferencing application screen 103). For example, the panoramic image 203, the speaker image 204, and the application screen 103 are combined so that the panoramic image 203 and the speaker image 204 are on the left and the remote conferencing application screen 103 is on the right (hereinafter referred to as the combined image 105). The application screen is an example of screen information (described later) displayed by each application such as the remote conferencing application. Since the process in (3) is executed repeatedly, the combined image 105 becomes a video (hereinafter referred to as the combined image video). The information recording application 41 also combines the audio synthesized with the combined image video to create a video with audio.

[0015] In this embodiment, an example is described in which the panoramic image 203, speaker image 204, and application screen 103 are combined. However, the information recording application 41 may save these separately and arrange them on the screen during playback.

[0016] (4) The information recording application 41 accepts editing work (cutting out unnecessary parts by the user) and completes the combined image video. The combined image video constitutes part of the recorded information.

[0017] (5) The information recording application 41 sends the created combined image video (with audio) to the storage service system 70 for storage.

[0018] (6) The information recording application 41 also extracts only the audio from the combined image video (it may also keep the audio before combining) and sends the extracted audio to the information processing system 50. The information processing system 50 sends the audio to the speech recognition service system 80, which converts the audio into text data. The text data also includes data on how many minutes after the start of recording the speaker spoke.

[0019] In the case of real-time text conversion, the meeting device 60 directly transmits audio to the information processing system 50. The meeting device 60 then transmits the text data to the information recording application 41 in real time.

[0020] (7) The information processing system 50 additionally stores text data in the storage service system 70 which stores the combined image video. The text data constitutes part of the recorded information.

[0021] Furthermore, the information processing system 50 can perform billing processing for users based on the services they have used. For example, billing is calculated based on the amount of text data, the file size of combined images and videos, processing time, etc.

[0022] In this way, the combined image video displays panoramic images of the surroundings including the user, as well as images of the speaker, and also shows the screen of the application used during the remote meeting, such as the remote conferencing application 42. When participants or non-participants of the remote meeting view the combined image video as meeting minutes, the atmosphere of the remote meeting is recreated with a sense of presence.

[0023] <Process Overview> Next, referring to Figure 2, we will explain the process by which the meeting device 60 extracts the display image showing the electronic whiteboard 2. Figure 2 is a diagram illustrating the process by which the meeting device 60 extracts the display image showing the electronic whiteboard 2.

[0024] (1) The user operates the electronic whiteboard 2, such as by writing on it.

[0025] (2) When the electronic whiteboard 2 detects an operation by the user, it transmits a message to the terminal device 10 indicating that the operation has started. As will be described later, the electronic whiteboard 2 and the terminal device 10 are pre-configured to be able to communicate with each other.

[0026] (3) When the terminal device 10 receives the start of operation, it sends an electronic whiteboard recording start request to the meeting device 60. The electronic whiteboard recording start request asks the meeting device 60 to extract the display device image 215 showing the electronic whiteboard 2.

[0027] (4) When the meeting device 60 receives a request to start recording the electronic whiteboard, it extracts a display device image 215 showing the electronic whiteboard 2 from the panoramic image 203 that captures the surroundings. The display device image 215 is transmitted to the terminal device 10 along with the speaker image 204 and the panoramic image 203. The terminal device 10 displays the display device image 215 so that the user can view the handwritten data etc. displayed on the electronic whiteboard 2.

[0028] In a similar manner, when the electronic whiteboard 2 detects that the user has finished operating on the electronic whiteboard 2, the electronic whiteboard 2 transmits a message indicating the end of operation to the terminal device 10. Upon receiving the message, the terminal device 10 sends a request to terminate the electronic whiteboard recording to the meeting device 60. Upon receiving the request to terminate the electronic whiteboard recording, the meeting device 60 terminates the process of extracting the display device image 215 from the panoramic image.

[0029] Therefore, the recording information creation system 100 of this embodiment extracts the display device image 215 when the electronic whiteboard 2 is operated, so that when new handwriting is done on the electronic whiteboard 2, the handwritten data can be displayed. When the handwriting is finished, the display of the display device image 215 can also be terminated.

[0030] The following are some ways in which the meeting device 60 can determine the orientation of the electronic whiteboard 2. • At the start of the meeting (or during the meeting), the user specifies a range for the panoramic image that shows the electronic whiteboard 2. • A specific image (such as a QR code) is displayed on the electronic whiteboard 2, and recognized from the panoramic image captured by the imaging unit of the meeting device 60. The QR code can be, for example, a QR code (registered trademark). The electronic whiteboard 2 outputs a specific sound, and the meeting device 60 recognizes the direction using its microphone. The shape of the electronic whiteboard 2 is learned through machine learning, and the meeting device or terminal device 10 recognizes it from the panoramic image 203.

[0031] <About Terminology> An application (app) is software developed and used for a specific function or purpose. Applications can be native apps or web apps. A web app (a cloud-based app provided by a cloud service) may also work in conjunction with a native app or web browser.

[0032] A running app refers to an app in its state from the time it is launched until it is closed. The app does not need to be active (the app in the foreground); it only needs to be running in the background.

[0033] The image information of the surroundings of a meeting device acquired by a meeting device is image information acquired by the meeting device by capturing the surrounding space (for example, a space of 180° to 360° in the horizontal direction), and refers to an image acquired by performing predetermined processing on the curved surface image information captured by the meeting device. The predetermined processing includes various processes for creating surrounding image information from the captured information, such as a flattening process for the captured curved surface image. In addition to the process of creating the surrounding image, the predetermined processing may also include a process of cutting out the speaker image, and a merging process that combines the surrounding image and the speaker image, etc. In this embodiment, the surrounding image is described using the term panoramic image. A panoramic image is an image with a field of view of approximately 180° to 360° in the horizontal direction. The meeting device does not have to capture a panoramic image with a single unit; multiple imaging devices with normal fields of view may be combined. Meeting devices are intended for use at locations such as tables for meetings and to monitor the surrounding environment. However, they may also be used for other purposes, such as monitoring (security, disaster prevention, etc.), supervision (childcare, elder care, etc.), and on-site situation analysis (solutions, marketing, etc.).

[0034] Recorded information refers to information recorded by the information recording application 41, which is stored and made viewable as information linked to the identification information of a particular meeting, and includes, for example, the following information: Video information created based on screen information displayed by the selected application (such as a remote conferencing application) and panoramic images of the device's surroundings acquired by the device. • Audio information acquired and synthesized during a meeting using a remote conferencing application (terminal device) and a meeting device at the local office. • Text information obtained by converting the acquired audio into text. Other data and images related to the meeting. For example, document files used during the meeting, additional notes, translated text data, images and stroke data created using a cloud-based electronic whiteboard service during the meeting, etc. For example, if the information recording application 41 records the screen of a remote meeting application or the proceedings of a meeting at a local office, the recorded information may become the minutes of the meeting that took place. The minutes are just one example of the recorded information, and the name of the recorded information changes depending on the content of the remote meeting or what was done at the local office. For example, it could be called a record of communication, a record of the local office's status, etc. The recorded information also includes multiple file formats, such as video files (combined video images, etc.), audio files, text data (text data from speech recognition), document files, image files, and tabular files. Since the files are related to each other in relation to the meeting identification information, they can be viewed chronologically, either all at once or selectively.

[0035] A tenant is a group of users (companies, local governments, and some of these organizations) that have contracted to receive services from a service provider. The creation of record information and its conversion to text data in this embodiment are performed because the tenant has a contract with the service provider.

[0036] Remote communication refers to communication with someone located at a physically distant location, using software and terminal devices to communicate via voice and video. One example of remote communication is a remote conference, which may also be called a meeting, discussion, consultation, contract application, gathering, assembly, seminar, workshop, study group, seminar, training session, etc.

[0037] A base of operations refers to a place that serves as the foundation for activities. An example of a base of operations is a conference room. A conference room is a room set up primarily for the purpose of holding meetings. A base of operations can also be a home, a reception area, a store, a warehouse, or an outdoor work site, as long as there is a place or space where terminal equipment or devices can be installed.

[0038] Speech refers to human speech sounds and ambient sounds, while speech data is the data representation of speech. However, in this embodiment, we will not strictly distinguish between the two.

[0039] The display device can be any device that displays information and allows users to operate it or approach it for explanation. In this embodiment, the display device is referred to as "electronic blackboard 2." An electronic blackboard may also be called an electronic whiteboard, electronic information board, etc. A projector is known as a device equivalent to electronic blackboard 2. When electronic blackboard 2 is a projector, the target of the cropping is not the projector itself, but the image projected by the projector. In this case, it becomes difficult to detect the direction by a specific sound, but the direction of the projected image can be determined by other methods. In addition, the display device may be a digital signage, television, display, multifunction printer, video conferencing terminal, etc.

[0040] The term "user using a display device" includes not only the user actually using the display device, but also the likelihood of the user using the display device. The likelihood of the user using the display device includes situations such as the user approaching the display device or holding a pen in their hand.

[0041] <Example System Configuration> Next, the system configuration of the recording information creation system 100 will be explained with reference to Figure 3. Figure 3 shows an example of the configuration of the recording information creation system 100. In Figure 3, one of several locations where remote meetings are held (local location 102) is shown, and the terminal device 10 at local location 102 communicates with the information processing system 50, the storage service system 70, and the remote meeting service system 90 via the network. A meeting device 60 and an electronic whiteboard 2 are also located at local location 102, and the terminal device 10 is connected to the meeting device 60 via a USB cable or the like for communication.

[0042] Terminal device 10 operates at least an information recording application 41 and a remote conferencing application 42. The remote conferencing application 42 can communicate with terminal devices 10 at other locations 101 via a remote conferencing service system 90 on the network, allowing users at each location to hold meetings remotely. The information recording application 41 creates recording information for remote meetings conducted by the remote conferencing application 42 using the functions of the information processing system 50 and the meeting device 60.

[0043] In this embodiment, we describe an example of creating recording information during a remote meeting, but the meeting does not have to be one that communicates with remote locations. In other words, the meeting may be one in which only participants from a single location are present. In this case, only the audio collected by the meeting device 60 is saved without synthesis, and there is no change in the processing of the information recording application 41.

[0044] The terminal device 10 has a built-in (or external) camera with a normal field of view that captures an image of the front, including the user 107 operating the terminal device 10. A normal field of view is an image that is not a panoramic image, but in this embodiment, it is mainly a planar image that is not curved like a 360-degree image. The terminal device 10 also has a built-in (or external) microphone that collects ambient sounds such as those of the user operating the terminal device 10. Therefore, the user can conduct a conventional remote conference using the remote conference application 42 without being aware of the information recording application 41. The information recording application 41 and the meeting device 60 do not affect the remote conference application 42, except for the increased processing load on the terminal device 10.

[0045] The information recording application 41 is an application that records information by communicating with the meeting device 60 and creating recorded information. The meeting device 60 is a meeting device equipped with a panoramic image capture device, a microphone, and a speaker. The camera on the terminal device 10 can only capture a limited area in front of it, but the meeting device 60 can capture the entire surrounding area (not necessarily all around) of the meeting device 60. The meeting device 60 can always keep multiple participants 106, as shown in Figure 3, within its field of view.

[0046] In addition, the meeting device 60 extracts speaker images from the panoramic image, and combines the audio acquired by the meeting device 60 with the audio output by the terminal device 10 (including the audio received by the remote conferencing application 42). The meeting device 60 is not limited to being placed on a table or stand; it can be placed anywhere within the base 102. Since the meeting device 60 can capture a full-spherical image, it can be placed on the ceiling, for example. The meeting device 60 only needs to be able to capture a hemispherical image. Furthermore, the meeting device 60 can be installed at other bases or at any of the bases.

[0047] The information recording application 41 displays a list of applications running on the terminal device 10, performs image synthesis (creation of combined image videos) for the recorded information described above, plays the combined image videos, accepts editing requests, etc. The information recording application 41 also displays a list of remote meetings that have been held or are scheduled to be held. The list of remote meetings is used for information related to the recorded information, allowing the user to link remote meetings with the recorded information.

[0048] The remote conferencing application 42 is an application that enables a terminal device to communicate remotely with other terminal devices at other locations 101 through communication connections, transmission and reception of images and audio, display of images, and output of audio. The remote conferencing application can also be called a remote communication application or a remote information sharing application.

[0049] The information recording application 41 and the remote conferencing application 42 may be either web applications or native applications. A web application is an application in which a program on a web server and a program on a web browser or a native application work together to process data, and does not require installation on the terminal device 10. A native application is an application that is installed on the terminal device 10 and used by it. In this embodiment, both will be described as native applications.

[0050] The terminal device 10 may be a general-purpose information processing device equipped with communication functions, such as a PC (Personal Computer), smartphone, or tablet terminal. Other examples of terminal devices 10 include an electronic whiteboard 2, a game console, a PDA (Personal Digital Assistant), a wearable PC, a car navigation system, industrial machinery, medical equipment, or network-connected home appliances. The terminal device 10 only needs to be a device on which at least the information recording application 41 and the remote conferencing application 42 can run.

[0051] The electronic whiteboard 2 displays data handwritten on a touch panel using input methods such as a pen or finger. The electronic whiteboard 2 can communicate with terminal devices 10 etc. via wired or wireless connection and can capture the screen displayed by terminal devices 10 and display it on the display. The electronic whiteboard 2 can convert handwritten data into text data and share information displayed on the display with electronic whiteboards 2 at other locations. The electronic whiteboard 2 may also be a simple whiteboard without a touch panel on which a projector projects images. Alternatively, the electronic whiteboard 2 may be a tablet device, notebook PC (Personal Computer), PDA (Personal Digital Assistant), game console, etc., equipped with a touch panel.

[0052] The electronic whiteboard 2 can communicate with the information processing system 50. For example, after being powered on, the electronic whiteboard 2 can receive information from the information processing system 50 by polling it. The electronic whiteboard 2 and the meeting device 60, which can capture images of the electronic whiteboard 2, operate as a display system.

[0053] The information processing system 50 consists of one or more information processing devices located on a network. The information processing system 50 has one or more server applications that perform processing in cooperation with the information recording application 41, and infrastructure services. These server applications manage lists of remote meetings, manage recorded information from remote meetings, manage various settings and storage paths, etc. The infrastructure services handle user authentication, contracts, billing, etc.

[0054] Furthermore, all or part of the functions of the information processing system 50 may reside in a cloud environment or in an on-premises environment. The information processing system 50 may consist of multiple server devices or a single information processing device. For example, server applications and infrastructure services may be provided by separate information processing devices, and furthermore, there may be separate information processing devices for each function within the server application. The information processing system 50 may be integrated with the storage service system 70 and the speech recognition service system 80 described below.

[0055] The storage service system 70 is a storage means on a network and provides a storage service that accepts the storage of files, etc. Examples of storage service systems 70 include OneDrive®, Google Workspace®, and Dropbox®. The storage service system 70 may also be an on-premises NAS (Network Attached Storage).

[0056] The speech recognition service system 80 provides a service that performs speech recognition on voice data and converts it into text data. The speech recognition service system 80 may be a general-purpose commercial service or it may be part of the functions of the information processing system 50. In addition, the speech recognition service system 80 may be configured and used differently for each user, tenant, or meeting.

[0057] <Example Hardware Configuration> Referring to Figure 4, the hardware configuration of the information processing system 50 and terminal device 10 according to this embodiment will be described.

[0058] <<Information Processing Systems and Terminal Devices>> Figure 4 shows the hardware configuration of an example of the information processing system 50 and terminal device 10 according to this embodiment. As shown in Figure 4, the information processing system 50 and terminal device 10 are built using a computer and include a CPU 501, ROM 502, RAM 503, HD (Hard Disk) 504, HDD (Hard Disk Drive) controller 505, display 506, external device connection I / F (Interface) 508, network I / F 509, bus line 510, keyboard 511, pointing device 512, optical drive 514, and media I / F 516.

[0059] Of these, the CPU 501 controls the operation of the entire information processing system 50 and terminal device 10. The ROM 502 stores programs used to drive the CPU 501, such as IPL. The RAM 503 is used as the work area for the CPU 501. The HD 504 stores various data such as programs. The HDD controller 505 controls the reading or writing of various data to the HD 504 according to the control of the CPU 501. The display 506 displays various information such as cursors, menus, windows, characters, or images. The external device connection I / F 508 is an interface for connecting various external devices. In this case, external devices include, for example, USB (Universal Serial Bus) memory and printers. The network I / F 509 is an interface for data communication using a network. The bus line 510 is an address bus, data bus, etc., for electrically connecting each component such as the CPU 501 shown in Figure 4.

[0060] The keyboard 511 is a type of input means equipped with multiple keys used for inputting characters, numbers, or various instructions. The pointing device 512 is a type of input means used for selecting and executing various instructions, selecting processing targets, moving the cursor, etc. The optical drive 514 controls the reading or writing of various data to the optical storage medium 513, which is an example of a removable recording medium. The optical storage medium 513 may be a CD, DVD, Blu-ray (registered trademark), etc. The media I / F 516 controls the reading or writing (storage) of data to the recording medium 515, such as flash memory.

[0061] <<Meeting Devices>> The hardware configuration of the meeting device 60 will be explained using Figure 5. Figure 5 is an example of a hardware configuration diagram for a meeting device 60 capable of capturing 360° video. In the following, the meeting device 60 is assumed to be a device that uses an image sensor to capture 360° video around the device at a predetermined height, but it may have one or more image sensors. Furthermore, it does not necessarily have to be a dedicated device; a 360° video capture unit can be attached to a PC, digital camera, smartphone, etc., to achieve substantially the same function.

[0062] As shown in Figure 5, the meeting device 60 consists of an imaging unit 601, an image processing unit 604, an imaging control unit 605, a microphone 608, a sound processing unit 609, a CPU (Central Processing Unit) 611, a ROM (Read Only Memory) 612, an SRAM (Static Random Access Memory) 613, a DRAM (Dynamic Random Access Memory) 614, an operation unit 615, an external device connection interface 616, a communication unit 617, an antenna 617a, a sound sensor 618, and a recessed terminal 621 for Micro USB.

[0063] Of these, the imaging unit 601 includes a wide-angle lens (a so-called fisheye lens) 602 with a 360° field of view for forming a hemispherical image, and an image sensor 603 provided corresponding to each wide-angle lens. The image sensor 603 has an image sensor such as a CMOS (Complementary Metal Oxide Semiconductor) sensor or a CCD (Charge Coupled Device) sensor that converts the optical image from the fisheye lens 602 into electrical signal image data and outputs it, a timing generation circuit that generates horizontal or vertical synchronization signals and pixel clocks for this image sensor, and a group of registers in which various commands and parameters necessary for the operation of this image sensor are set. The imaging unit 601 may also be a 360° camera, and is an example of an imaging means capable of imaging 360° around the meeting device 60.

[0064] Each image sensor 603 of the imaging unit 601 is connected to the image processing unit 604 via a parallel I / F bus. On the other hand, the image sensor 603 of the imaging unit 601 is connected to the imaging control unit 605 via a serial I / F bus (such as an I2C bus). The image processing unit 604, the imaging control unit 605, and the sound processing unit 609 are connected to the CPU 611 via bus 610. Furthermore, ROM 612, SRAM 613, DRAM 614, operation unit 615, external device connection I / F 616, communication unit 617, and sound sensor 618 are also connected to bus 610.

[0065] The image processing unit 604 receives image data output from the image sensor 603 via a parallel I / F bus, performs predetermined processing on each image data, and creates panoramic images and speaker images from the fisheye video. Furthermore, the image processing unit 604 combines the panoramic image and speaker images, etc., to output a single video.

[0066] The imaging control unit 605 generally uses the I2C bus to set commands and other information in the registers of the image sensor 603, with the imaging control unit 605 acting as the master device and the image sensor 603 as the slave device. It receives necessary commands and other information from the CPU 611. The imaging control unit 605 also uses the I2C bus to acquire status data and other information from the registers of the image sensor 603 and send it to the CPU 611.

[0067] Furthermore, the imaging control unit 605 instructs the image sensors 603a and 603b to output image data when the image capture start button on the operation unit 615 is pressed or when an image capture start instruction is received from the PC. Depending on the meeting device 60, there may also be functions to display a preview or video on a display (for example, the display of a PC or smartphone). In this case, the image data output from the image sensor 603 is performed continuously at a predetermined frame rate (frames / minute).

[0068] Furthermore, as will be described later, the imaging control unit 605 also functions as a synchronization control means that works in cooperation with the CPU 611 to synchronize the output timing of image data from the image sensor 603. In this embodiment, the meeting device 60 is not provided with a display, but a display unit may be provided.

[0069] Microphone 608 converts sound into sound (signal) data. Sound processing unit 609 receives the audio data output from microphone 608 through the I / F bus and performs predetermined processing on the audio data.

[0070] The CPU 611 controls the overall operation of the meeting device 60 and performs necessary processing. The ROM 612 stores various programs for the CPU 611. The SRAM 613 and DRAM 614 are work memories that store programs executed by the CPU 611 and data in progress. In particular, the DRAM 614 stores image data in progress and processed equirectangular projection image data from the image processing unit 604.

[0071] The control unit 615 is a collective term for the control buttons, such as the image capture start button 615a. By operating the control unit 615, the user can start imaging and recording, turn the power on / off, establish a communication connection, and input various settings such as imaging modes and imaging conditions.

[0072] The external device connection interface 616 is an interface for connecting various external devices. In this case, external devices include, for example, a PC (Personal Computer), a display, a projector, an electronic whiteboard, etc. The external device connection interface 616 may also be equipped with, for example, a USB terminal, an HDMI® terminal, etc. Video data and image data stored in the DRAM 614 are transmitted to an external terminal via this external device connection interface 616 or recorded on external media. Alternatively, multiple external device connection interfaces 616 may be used to transmit and record panoramic images captured by the meeting device 60 to a PC via USB, while simultaneously acquiring video (for example, screen information to be displayed in a remote conferencing application) from the PC to the meeting device 60, and then transmitting it from the meeting device 60 via HDMI® to other external devices (displays, projectors, electronic whiteboards, etc.) for display. The communication unit 617 may communicate with a cloud server via the internet using wireless communication technology such as Wi-Fi through the antenna 617a provided on the meeting device 60, and transmit the stored video data and image data to the cloud server. Alternatively, the communication unit 617 may communicate with nearby devices using short-range wireless communication technology such as BLE (Bluetooth Low Energy; registered trademark) or NFC.

[0073] The audio sensor 618 is a sensor that acquires 360° audio information in order to determine from which direction loud audio is being input within a 360° radius (horizontal plane) around the meeting device 60. The audio processing unit 609 identifies the direction of the strongest sound based on the input 360° audio parameters and outputs the audio input direction within the 360°.

[0074] Furthermore, other sensors (such as compass / accelerometers or GPS) can be used to calculate direction, position, angle, acceleration, etc., and these can be used for image correction or to add location information.

[0075] The image processing unit 604 also performs the following processing:

[0076] The CPU 611 creates a panoramic image in the following way: The CPU 611 performs predetermined camera image processing, such as Bayer conversion (RGB interpolation processing), on the RAW data input from the image sensor that inputs a spherical image to create a fisheye image (an image consisting of curved images). Furthermore, it performs planarization processing, such as DeWarp processing (distortion correction processing), on the created fisheye image (a curved image) to create a panoramic image (an image consisting of flat images) that captures 360° around the meeting device 60.

[0077] The CPU 611 creates speaker images in the following way: The CPU 611 creates speaker images by extracting the speaker from a panoramic image (flat image) that captures the surrounding 360°. The CPU 611 uses the direction of the audio input, identified from the 360° output by the audio sensor 618 and the audio processing unit 609, as the direction of the speaker, and extracts the speaker image from the panoramic image. In this case, the method for extracting a person's image from the direction of the audio input is to cut out 30° around the audio direction identified from the 360°, and then perform face detection within that area to extract the image. The CPU 611 further identifies speaker images for a specific number of people (e.g., 3 people) who have spoken most recently from the extracted speaker images.

[0078] The panoramic image and the images of one or more speakers may be sent individually to the information recording application 41, or the meeting device 60 may create a single image from them and send it to the information recording application 41. In this embodiment, the panoramic image and the images of one or more speakers are sent individually from the meeting device 60 to the information recording application 41.

[0079] Figure 6 illustrates the imaging range of the meeting device 60. As shown in Figure 6(a), the meeting device 60 images a 360° range in the horizontal direction. As shown in Figure 6(b), the meeting device 60 has an imaging range that is 0° horizontal to the height of the meeting device 60, and a predetermined angle above and below it.

[0080] Figure 7 illustrates the extraction of panoramic and speaker images. As shown in Figure 7, the image captured by the meeting device 60 forms a part 110 of a sphere and therefore has a three-dimensional shape. As shown in Figure 6(b), the meeting device 60 divides the field of view into predetermined angles vertically and horizontally and performs perspective projection transformation. By performing perspective projection transformation seamlessly across the entire 360° horizontal direction, a predetermined number of planar images are obtained, and by stitching these predetermined number of planar images together horizontally, a panoramic image 111 is obtained. The meeting device 60 also performs face detection from the panoramic image within a predetermined range centered on the direction of sound, and creates speaker images 112 by extracting 15° to the left and right (30° in total) from the center of the face.

[0081] <<Electronic Whiteboard>> Figure 8 is a hardware configuration diagram of the electronic whiteboard 2. As shown in Figure 8, the electronic whiteboard 2 is equipped with a CPU (Central Processing Unit) 401, ROM (Read Only Memory) 402, RAM (Random Access Memory) 403, SSD (Solid State Drive) 404, network I / F 405, and external device connection I / F (Interface) 406.

[0082] Of these, the CPU 401 controls the overall operation of the electronic whiteboard 2. The ROM 402 stores programs used to drive the CPU 401, such as the CPU 401 and the IPL (Initial Program Loader). The RAM 403 is used as the work area for the CPU 401. The SSD 404 stores various data, such as programs for the electronic whiteboard 2. The network I / F 405 controls communication with the communication network. The external device connection I / F 406 is an interface for connecting various external devices. In this case, external devices include, for example, a USB (Universal Serial Bus) memory 430 and external devices (microphone 440, speaker 450, camera 460).

[0083] Furthermore, the electronic whiteboard 2 is equipped with a capture device 411, a GPU 412, a display controller 413, a contact sensor 414, a sensor controller 415, an electronic pen controller 416, a short-range communication circuit 419, and an antenna 419a for the short-range communication circuit 419, a power switch 422, and selection switches 423.

[0084] Of these, the capture device 411 displays video information as a still image or video on the display of an external PC (Personal Computer) 470. The GPU (Graphics Processing Unit) 412 is a semiconductor chip specializing in graphics. The display controller 413 controls and manages screen display in order to output the output image from the GPU 412 to the display 480, etc. The contact sensor 414 detects when an electronic pen 490 or the user's hand H, etc., touches the display 480. The sensor controller 415 controls the processing of the contact sensor 414. The contact sensor 414 performs coordinate input and coordinate detection using an infrared blocking method. This method of coordinate input and coordinate detection involves two light-emitting and receiving devices installed at both ends of the upper side of the display 480 emitting multiple infrared rays parallel to the display 480, and receiving the light that is reflected by reflective members provided around the display 480 and returns along the same optical path as the light emitted by the light-receiving element. The contact sensor 414 outputs the infrared ID emitted by two light-receiving devices that are blocked by an object to the sensor controller 415, which then identifies the coordinate position of the object's contact. The electronic pen controller 416 communicates with the electronic pen 490 to determine whether the pen tip or the pen end has touched the display 480. The near-field communication circuit 419 is a communication circuit such as NFC (Near Field Communication) or Bluetooth (registered trademark). The power switch 422 is a switch for turning the power of the electronic whiteboard 2 ON / OFF. The selection switches 423 are a group of switches for adjusting, for example, the brightness and color of the display 480.

[0085] Furthermore, the electronic whiteboard 2 is equipped with a bus line 410. The bus line 410 is an address bus, data bus, etc., for electrically connecting each component, such as the CPU 401 shown in Figure 8.

[0086] Furthermore, the contact sensor 414 is not limited to an infrared blocking type; it may also be a capacitive type touch panel that identifies the contact position by detecting changes in capacitance. The contact sensor 414 may also be a resistive type touch panel that identifies the contact position by voltage changes between two opposing resistive films. The contact sensor 414 may also be an electromagnetic induction type touch panel that identifies the contact position by detecting electromagnetic induction caused by contact between an object and the display unit. In addition, the contact sensor 414 may use various other detection means. Moreover, the electronic pen controller 416 may determine whether there is a touch not only on the tip and end of the electronic pen 490, but also on the part of the electronic pen 490 held by the user and other parts of the electronic pen.

[0087] <About the features> Next, with reference to Figure 9, the functional configuration of the recording information creation system 100 will be described. Figure 9 is an example of a functional block diagram that explains the functions of the terminal device 10, meeting device 60, and information processing system 50 in the recording information creation system 100 by dividing them into blocks.

[0088] <<Terminal device>> The information recording application 41 operating on the terminal device 10 includes a communication unit 11, an operation reception unit 12, a display control unit 13, an application screen acquisition unit 14, an audio acquisition unit 15, a device communication unit 16, a recording control unit 17, an audio data processing unit 18, a recording playback unit 19, an upload unit 20, an editing processing unit 21, an analysis unit 22, a cut-out request unit 23, and a display device recognition unit 24. Each of these parts of the terminal device 10 is a function or means of functioning that is realized by any of the components shown in Figure 4 operating according to instructions from the CPU 501 in accordance with the information recording application 41 deployed from the HD 504 to the RAM 503. The terminal device 10 also has a storage unit 1000 constructed by the HD 504 etc. shown in Figure 4. An information storage unit 1001 is constructed in the storage unit 1000.

[0089] The communication unit 11 communicates various types of information with the information processing system 50 via the network. For example, the communication unit 11 receives a list of remote meetings from the information processing system 50 and sends voice data recognition requests to the information processing system 50. The communication unit 11 also receives operation start and operation end notifications from the electronic whiteboard 2.

[0090] The display control unit 13 displays various screens that serve as the user interface in the information recording application 41 according to the screen transitions set in the information recording application 41. The operation reception unit 12 receives various operations on the information recording application 41.

[0091] The application screen acquisition unit 14 acquires screen information displayed by the application selected by the user, or screen information of the desktop screen, etc., from the OS (Operating System). If the application selected by the user is the remote conferencing application 42, the screen generated by the remote conferencing application 42 is obtained (images of terminal device users captured by terminal device cameras at each location, images of shared documents, images including participant icons and participant names, etc.). The screen information displayed by the application (application screen) is information displayed as a window by the running application and acquired as an image by the information recording application. The application window is drawn as an area of ​​the window within the entire desktop image and displayed on a monitor, etc. The screen information displayed by the application can be acquired by other applications (information recording applications, etc.) as an image file or a video file composed of a series of images via the OS (Operating System) API or the API of the application displaying it. Similarly, the screen information of the desktop screen is information composed of images of the desktop screen generated by the OS, and can be acquired as an image file or video file via the OS API. The format of these image files may be bitmap, PNG, or other formats. The video file format can be MP4 or other formats.

[0092] The audio acquisition unit 15 acquires audio output from the microphone or earphones of the terminal device 10 (including audio data received from the remote conferencing application 42 during a remote conference). The audio acquisition unit 15 can acquire audio even if the output audio is muted. The user does not need to perform any operations such as selecting the remote conferencing application 42 regarding audio data; the audio acquisition unit 15 can acquire the audio that the terminal device 10 can output via the OS or application's API (Application Interface). This also allows the audio data received by the remote conferencing application 42 from other locations 101 to be acquired. If the remote conferencing application 42 is not running or a remote conference is not in progress, the information recording application 41 may not be able to acquire audio data. Note that the audio acquired by the audio acquisition unit 15 may not include the audio collected by the terminal device 10, but only the output audio data. This is because the meeting device 60 separately collects audio from the locations.

[0093] The device communication unit 16 communicates with the meeting device 60 using a USB cable or the like. The device communication unit 16 may communicate with the meeting device 60 using wireless LAN, Bluetooth (registered trademark), etc. The device communication unit 16 receives panoramic images and speaker images from the meeting device 60 and transmits the audio data acquired by the audio acquisition unit 15 to the meeting device 60. The device communication unit 16 receives the audio data synthesized by the meeting device 60.

[0094] The recording control unit 17 combines the panoramic image and speaker image received by the device communication unit 16, and the application screen acquired by the application screen acquisition unit 14, to create a combined image. The combination of the panoramic image and speaker image may be performed by the meeting device 60. Alternatively, the video consisting of each image, such as the panoramic image, speaker image, application screen, and the image composed of the panoramic image and speaker image, may be stored as separate video files in the storage service system 70. In that case, the panoramic video, speaker video, application screen video, and combined video of the panoramic image and speaker image may be recalled and displayed on a single display screen when viewed.

[0095] The audio data processing unit 18 either extracts the audio data combined into the combined image video, or requests the information processing system 50 to convert the synthesized audio data received from the meeting device 60 into text data.

[0096] The recording and playback unit 19 plays back the combined image video. The combined image video is stored in the terminal device 10 during recording and then uploaded to the information processing system 50.

[0097] When the remote conference ends, the upload unit 20 transmits the combined image and video to the information processing system 50.

[0098] The editing processing unit 21 performs editing of the combined image video (such as deleting parts or splicing them together) in response to user operations.

[0099] The analysis unit 22 detects the two-dimensional code contained in the panoramic image and analyzes the two-dimensional code to obtain information contained in the two-dimensional code, such as device identification information.

[0100] When the extraction request unit 23 receives an operation start notification from the electronic whiteboard 2, it requests the meeting device 60 to start recording the electronic whiteboard image 215 (extraction request). Furthermore, when the extraction request unit 23 receives an operation end notification from the electronic whiteboard 2, it requests the meeting device 60 to end the electronic whiteboard recording (extraction end) of the display device image 215.

[0101] The display device recognition unit 24 learns the shape (circumscribed rectangle) of the electronic whiteboard 2 using machine learning and detects the electronic whiteboard 2 from the panoramic image. The display device recognition unit 24 may also recognize the shape simply by pattern matching without using machine learning.

[0102] Figure 10 shows the video recording information stored in the information storage unit 1001. The video recording information includes items such as meeting ID, recording ID, update date and time, title, upload, and storage location. When a user logs in to the information processing system 50, the information recording application 41 downloads meeting information from the meeting information storage unit 5001 of the information processing system 50. The meeting ID and other information included in the meeting information are reflected in the video recording information. The video recording information in Figure 10 is held by a terminal device 10 operated by a particular user.

[0103] The meeting ID is identification information that identifies a remote meeting that has been held. The meeting ID is assigned when the remote meeting is scheduled in the meeting management system 9, or it is assigned by the information processing system 50 upon request from the information recording application 41. The meeting management system 9 is a system for registering meeting and remote meeting schedules, URLs (meeting links) for starting remote meetings, and reservation information for equipment used in meetings. It is a scheduler, etc., that is connected to from the terminal device 10 via the network. The meeting management system 9 can also transmit the registered schedules, etc., to the information processing system 50.

[0104] The recording ID is identification information that identifies the combined video image recorded during a remote meeting. The recording ID is assigned by the meeting device 60, but it may also be assigned by the information recording application 41 or the information processing system 50. The same meeting ID may be assigned to a different recording ID if the recording of the remote meeting was stopped midway through but resumed for some reason.

[0105] • The update date and time is the date and time when the combined image / video was updated (recording ended). If the combined image / video was edited, it will be the date and time of editing.

[0106] The title is the name of the meeting. This may be set when registering the meeting in the meeting management system 9, or it may be set arbitrarily by the user.

[0107] • Upload indicates whether or not the combined image video has been uploaded to the information processing system 50.

[0108] The storage location indicates the location (URL or file path) where the combined image / video and text data are stored in the storage service system 70. Therefore, users can view the uploaded combined image / video at their discretion. Note that the combined image / video and text data are saved with separate filenames, for example, following the URL.

[0109] <<Meeting Devices>> Let's return to Figure 9 for explanation. The meeting device 60 includes a terminal communication unit 61, a panoramic image generation unit 62, a speaker image generation unit 63, a sound collection unit 64, a speech synthesis unit 65, a display device image generation unit 66, and a display device identification unit 67. Each of these parts of the meeting device 60 is a function or means of functioning, realized by any of the components shown in Figure 5 operating according to instructions from the CPU 611 in accordance with a program expanded from the ROM 612 to the DRAM 614.

[0110] The terminal communication unit 61 communicates with the terminal device 10 using a USB cable or the like. The terminal communication unit 61 does not only need to be connected by a wired cable, but it is also acceptable if it can communicate with the terminal device 10 via wireless LAN, Bluetooth (registered trademark), etc.

[0111] The panoramic image generation unit 62 creates a panoramic image. The speaker image generation unit 63 creates a speaker image. The methods for creating these are explained in Figures 6 and 7.

[0112] The sound collection unit 64 converts the audio signal acquired by the microphone of the meeting device 60 into audio data (digital). This allows the content spoken by users and participants at the terminal device 10 to be collected.

[0113] The speech synthesis unit 65 synthesizes the speech transmitted from the terminal device 10 with the speech collected by the sound collection unit 64. This combines speech spoken at other locations 101 with speech spoken at the local location 102.

[0114] The display device image generation unit 66 starts extracting the display device image 215 from the panoramic image in response to a request to start electronic whiteboard recording (extraction request) from the terminal device 10, and stops extracting the display device image 215 in response to a request to end electronic whiteboard recording (extraction end request).

[0115] The display device identification unit 67 identifies the orientation of the electronic whiteboard 2 in the panoramic image based on information about the display device, such as the orientation of the electronic whiteboard 2, detection of a two-dimensional code, detection of sound of a specific frequency, or the shape of the electronic whiteboard 2. The display device identification unit 67 may also acquire display device identification information by performing voice recognition. Furthermore, this identification of the display device may be performed by either the terminal device 10 or the meeting device 60. If the terminal device 10 performs the identification, the display device identification unit 67 retains the orientation of the electronic whiteboard 2 transmitted from the terminal device 10.

[0116] <<Information Processing System>> The information processing system 50 includes a communication unit 51, an authentication unit 52, a screen generation unit 53, a communication management unit 54, an equipment management unit 55, and a text conversion unit 56. Each of these units in the information processing system 50 is a function or means of functioning that is realized by any of the components shown in Figure 4 operating according to instructions from the CPU 501 that follow a program deployed from the HD 504 to the RAM 503. The information processing system 50 also has a storage unit 5000 constructed from the HD 504 and other components shown in Figure 4. The storage unit 5000 includes a conference information storage unit 5001, a recording information storage unit 5002, a mapping information storage unit 5003, and an account information storage unit 5004.

[0117] The communication unit 51 sends and receives various types of information with the terminal device 10. For example, the communication unit 51 may send a list of remote meetings to the terminal device 10 or receive a voice data recognition request from the terminal device 10.

[0118] The authentication unit 52 authenticates the user operating the terminal device 10. The authentication unit 52 authenticates the user, for example, by checking whether the authentication information (user ID and password) included in the authentication request received by the communication unit 51 matches the authentication information it has previously stored. The authentication information may be an IC card number, biometric authentication information such as a face or fingerprint, etc. The authentication unit 52 may also authenticate using an external authentication system or an authentication method such as OAuth.

[0119] The screen generation unit 53 generates screen information to be displayed by the terminal device 10. When the terminal device 10 runs a native application, the screen information is held by the terminal device 10, and the information to be displayed is transmitted in XML or the like. When the terminal device 10 runs a web application, the screen information is created using HTML, XML, CSS (Cascade Style Sheet), and JavaScript (registered trademark), etc.

[0120] The communication management unit 54 retrieves information about remote meetings from the meeting management system 9 using each user's account or a system account assigned to the information processing system 50. The communication management unit 54 stores the meeting information of scheduled meetings in the meeting information storage unit, associating it with the meeting ID. The communication management unit 54 also retrieves meeting information that users belonging to the tenant have viewing rights to. Since each meeting is assigned a meeting ID, the remote meeting and its recorded information are associated using the meeting ID.

[0121] When the equipment management unit 55 receives equipment identification information from the terminal device 10, it associates the conference ID with the electronic whiteboard 2 and meeting device 60 used in the conference and stores the associated information in the associated information storage unit 5003. Since combined image videos are also associated with the conference ID, handwritten data and combined image videos are also associated. Furthermore, when recording is finished (when the conference ends), the equipment management unit 55 deletes the association from the associated information storage unit 5003.

[0122] The text conversion unit 56 converts the voice data requested by the terminal device 10 to be converted to text data into text data using an external voice recognition service. The text conversion unit 56 may also perform the conversion itself.

[0123] Figure 11 shows an example of meeting information stored in the meeting information storage unit 5001, which is managed by the communication management unit 54. The communication management unit 54 can use the above account to obtain a list of remote meetings that the user belonging to the tenant has permission to view. Note that viewing permission may also be granted directly to the meeting information managed by the communication management unit 54 from the information recording application of the terminal device 10. Furthermore, the remote meeting information that a user belonging to the tenant has permission to view includes information on meetings created by the user and information on meetings that the user has been granted viewing permission for by other users. In this embodiment, remote meetings are used as an example, but the list of remote meetings also includes meetings held in only one conference room.

[0124] Meeting information is managed using a meeting ID, which is associated with the organizer ID, title (meeting name), start date and time, end date and time, electronic whiteboard 2, meeting device, etc. These are just examples of meeting information; meeting information may include other details as well.

[0125] The "Organizer ID" field indicates the organizer (host) of the meeting.

[0126] The title field should describe the content of the meeting, such as the meeting name or the agenda items from overseas.

[0127] The "Start Date and Time" field indicates the scheduled date and time when the meeting will begin.

[0128] The "End Date and Time" field indicates the scheduled end date and time of the meeting.

[0129] The item for Electronic Whiteboard 2 is the identification information for Electronic Whiteboard 2 associated with the meeting.

[0130] The "Meeting Device" field contains identification information for the meeting device used in the meeting.

[0131] As shown in Figures 10 and 11, the combined video footage recorded during the meeting is identified by the meeting ID.

[0132] The recording information stored in the recording information storage unit 5002 may be the same as in Figure 10. However, the information processing system 50 has a list of combined image videos recorded by all users belonging to the tenant. The storage location (path information such as the URL of the cloud storage system) may be entered by the user on the user settings screen of the information recording application 41 of the terminal device 10 and stored in the recording information storage unit 5002.

[0133] Figure 12 shows the mapping information stored in the mapping information storage unit 5003, which associates the meeting ID with the device identification information (electronic whiteboard 2 and meeting device 60). The mapping information is retained from the time the information recording application 41 transmits the device identification information to the information processing system 50 until the recording is terminated.

[0134] Figure 13 shows an example of account information stored in the account information storage unit 5004. The account information includes not only human users but also non-human users such as the electronic whiteboard 2 and the meeting device 60.

[0135] The user ID is identification information for users who may participate in the meeting, as well as for electronic whiteboard 2 and meeting devices.

[0136] • The type refers to the type of account for each user, electronic whiteboard 2, or meeting device 60.

[0137] • The names are the user's name, the names of the electronic whiteboard 2, and the meeting device 60.

[0138] • The email address is used for the user, the electronic whiteboard 2, and the meeting device, etc.

[0139] <<Electronic Whiteboard>> Figure 14 is a functional block diagram illustrating the functions of the electronic whiteboard 2 in blocks. The electronic whiteboard 2 includes a contact position detection unit 31, a drawing data generation unit 32, a data recording unit 33, a display unit 34, a code generation unit 35, a communication unit 36, an audio data generation unit 27, an operation detection unit 38, and a user detection unit 39. Each function of the electronic whiteboard 2 is realized by an instruction from the CPU 401, following a program deployed from the SSD 404 onto the RAM 403, in accordance with one of the components shown in Figure 8.

[0140] The contact position detection unit 31 detects the coordinates of the position where the electronic pen 490 makes contact with the contact sensor 414. The drawing data generation unit 32 obtains the coordinates of the contact point of the pen tip of the electronic pen 490 from the contact position detection unit 31. The drawing data generation unit 32 connects these coordinate points by interpolation to generate stroke data.

[0141] The display unit 34 displays handwritten data, strings converted from handwritten data, and menus for user operation on the display.

[0142] The data recording unit 33 stores handwritten data written on the electronic whiteboard 2, shapes converted from the handwritten data into circles, triangles, etc., stamps such as "completed", PC screens, files, etc., in the object information storage unit 3002. Handwritten data, strings (including shapes), images such as PC screens, files, etc., are treated as objects. For handwritten data, a set of stroke data becomes a single object according to temporal divisions due to interruptions in handwriting and spatial divisions due to differences in handwriting location.

[0143] The communication unit 36 ​​connects to Wi-Fi or LAN and communicates with the information processing system 50. The communication unit 36 ​​transmits object information to the information processing system 50 and receives object information stored in the information processing system 50 and displays it on the display 480. The communication unit 36 ​​also transmits the start or end of an operation to the terminal device 10.

[0144] The code generation unit 35 generates a two-dimensional code by encoding the device identification information of the electronic whiteboard 2 stored in the device information storage unit 3001 and information indicating that it is a device usable in a meeting into a two-dimensional pattern. The code generation unit 35 can also encode the device identification information of the electronic whiteboard 2 and information indicating that it is a device usable in a meeting into a barcode. The device identification information may be a serial number or a UUID. The device identification information may also be set by the user.

[0145] The audio data generation unit 27 generates audio data by sampling signals of a preset frequency at regular intervals, similar to PCM conversion. The audio data is converted to analog by the D / A converter of the speaker 450 and output from the speaker 450. The audio data generation unit 27 may also output device identification information in audio.

[0146] The operation detection unit 38 detects user operations on the electronic whiteboard 2. For example, the operation detection unit 38 detects the start or end of an operation when the contact position detection unit 31 detects that the electronic pen 490 or the user's hand H has come into contact with the display 480 (touch panel) (even if it is just close).

[0147] The user detection unit 39 detects when a user approaches or moves away from the electronic whiteboard 2 using an infrared sensor, ultrasonic sensor, camera, etc.

[0148] Furthermore, both the operation detection unit 38 and the user detection unit 39 function as detection units that detect when a user is using the electronic whiteboard 2.

[0149] Furthermore, the electronic whiteboard 2 has a storage unit 3000 constructed by the SSD 404 shown in Figure 8. The storage unit 3000 has a device information storage unit 3001 and an object information storage unit 3002.

[0150] Figure 15 shows the device identification information and other data stored in the device information storage unit 3001. • The device identification information is the identification information for electronic whiteboard 2. The IP address is the IP address that other devices use to connect to the electronic whiteboard 2 via the network. • The password is used for authentication when other devices connect to the electronic whiteboard 2.

[0151] Figure 16 is a diagram illustrating the object information stored in the object information storage unit 3002. The object information manages the objects displayed by the electronic whiteboard 2. The object information is transmitted to the information processing system 50 and used as meeting minutes.

[0152] The meeting ID is the identification information for the meeting, notified by the information processing system 50.

[0153] • An object ID is an identifier that identifies an object.

[0154] • The type refers to the object's characteristics, including handwritten data, text, shapes, images, etc. Handwritten data is stroke data (a sequence of coordinate points). Text is a string of characters (character codes) converted from handwritten data. Strings are sometimes also called text data. Shapes are geometric shapes such as triangles and squares converted from handwritten data. Images can be in formats such as JPEG, PNG, or TIFF, imported from a PC or the internet, but the image format is not restricted and can be anything.

[0155] • One screen of the electronic whiteboard 2 is referred to as a page. The item on the page is its page number.

[0156] The coordinates indicate the position of an object relative to a predetermined origin on the electronic whiteboard 2. The object's position is, for example, the top-left vertex of its bounding rectangle. The coordinates are expressed, for example, in pixels on the display.

[0157] • The size is the width and height of the object's bounding rectangle.

[0158] <Screen transition> Next, with reference to Figures 17 to 20, several screens displayed by the terminal device 10 during a remote conference will be described. Figure 17 is the initial screen 200 displayed after login by the information recording application 41 running on the terminal device 10. The user of the terminal device 10 connects the information recording application 41 to the information processing system 50. When the user enters authentication information and successfully logs in, the initial screen 200 of Figure 17 is displayed.

[0159] The initial screen 200 has a fixed display button 201, a front view change button 202, a panoramic image 203, one or more speaker images 204a to 204c (hereinafter referred to as speaker image 204 when not distinguished), a recording start button 205, and a location registration button 207. If the meeting device 60 is already activated and capturing images of the surroundings when the user logs in, the panoramic image 203 and speaker images 204 created by the meeting device 60 are displayed on the initial screen 200. Therefore, the user can decide whether or not to start recording while viewing these images. If the meeting device 60 is not activated (not capturing images), the panoramic image 203 and speaker images 204 are not displayed.

[0160] The information recording application 41 may display speaker images 204 of all participants based on all faces detected from the panoramic image 203, or it may display only the speaker images 204 of the N people who have spoken most recently. Figure 17 shows an example where up to three speaker images 204 are displayed. There may be no speaker images 204 until a participant speaks (one is added each time a participant speaks), or three speaker images 204 of participants facing a predetermined direction may be displayed (they are swapped depending on the participant's speech).

[0161] If no one is speaking, such as immediately after the meeting device 60 is activated, a speaker image 204 will be created in a predetermined direction within the horizontal 360° (e.g., 0°, 120°, 240°). If a fixed display setting (described later) is configured, that setting will take precedence.

[0162] The fixed display button 201 is a button that allows the user to perform an operation to fix a certain area of ​​the panoramic image 203 as the speaker image 204 and zoom in on it.

[0163] The front view change button 202 is a button for the user to change the front view of the panoramic image 203 (since the panoramic image captures 360° horizontally, the right and left edges coincide). The user can slide the panoramic image 203 left or right with a pointing device to determine which participant is in the front view. The user's operation is sent to the meeting device 60, which changes the angle that is considered the front view out of the 360° horizontal direction to create a panoramic image and sends it to the terminal device 10.

[0164] The location registration button 207 is a button used by the user to identify the location (direction) of a device such as the electronic whiteboard 2.

[0165] When the user presses the recording start button 205, the information recording application 41 displays the recording settings screen 210 shown in Figure 18.

[0166] Figure 18 shows an example of the recording settings screen 210 displayed by the information recording application 41. On the recording settings screen 210, the user can set whether to record (include in recording) the panoramic image and speaker image created by the meeting device 60, as well as the desktop screen or the screen of the running application on the terminal device 10. If the information recording application 41 does not record either the panoramic image and speaker image, or the desktop screen or the screen of the running application, only the audio (audio output by the terminal device 10 + audio collected by the meeting device 60) will be recorded.

[0167] The camera toggle button 211 is a button that switches the recording of panoramic images and speaker images created by the meeting device 60 on and off. The camera toggle button 211 may also be configured to record panoramic images and speaker images individually.

[0168] The PC screen toggle button 212 is a button that switches the recording of the desktop screen of the terminal device 10 and the screen of an application running on the terminal device 10 on and off. When the PC screen toggle button 212 is on, the desktop screen is recorded.

[0169] If a user wants to record an app screen, they further select an app from the app selection field 213. The app selection field 213 displays the names of apps currently running on the terminal device 10 in a dropdown format. The user can select the app to record. This app name is obtained by the information recording app 41 from the OS. The information recording app 41 can display only apps that have a UI (screen) among the running apps. The remote conferencing app 42 may be included in the selected apps. Therefore, the information recording app 41 can also record the materials displayed by the remote conferencing app 42 and the participants at each location in video. In addition, the apps displayed in the dropdown are various apps running on the terminal device, such as presentation apps, word processing apps, spreadsheet apps, document creation and editing apps, cloud electronic whiteboard apps, and web browser apps. Therefore, the user can flexibly select the app screens to include in the combined video.

[0170] Furthermore, when recording on an app-by-app basis, users can select multiple apps. The information recording app 41 can record the screens of all selected apps.

[0171] If both the camera toggle button 211 and the PC screen toggle button 212 are turned off, the recording content confirmation window 214 will display "Only audio will be recorded." This audio consists of the audio output by the terminal device 10 (audio received by the remote conferencing application 42 from other locations 101) and the audio collected by the meeting device 60. In other words, if a remote conference is being held, the audio from the remote conferencing application 42 and the audio from the meeting device 60 will be saved regardless of image recording. However, the user may selectively stop saving the audio from the remote conferencing application 42 and the audio from the meeting device 60 in their settings.

[0172] Depending on the on / off combination of the camera toggle button 211 and the PC screen toggle button 212, the combined image video is recorded as follows. The combined image video is also displayed in real time in the recording content confirmation window 214.

[0173] If the camera toggle button 211 is on and the PC screen toggle button 212 is off, the panoramic image captured by the meeting device 60 and the speaker image will be displayed in the recording content confirmation window 214.

[0174] If the camera toggle button 211 is off and the PC screen toggle button 212 is on (screen is also selected), the recording content confirmation window 214 will display the desktop screen or the screen of the selected application.

[0175] When the camera toggle button 211 is on and the PC screen toggle button 212 is on, the recording content confirmation window 214 displays the panoramic image and speaker image captured by the meeting device 60, as well as the desktop screen and the screen of the selected application, side by side.

[0176] Therefore, there are cases where the panoramic image, speaker image, and app screen are not recorded, or where none of the panoramic image, speaker image, and app screen are recorded. However, in this embodiment, for convenience, the images created by the information recording app 41 are referred to as a combined image video.

[0177] Furthermore, the recording settings screen 210 has a checkbox 209 along with the message, "Automatically transcribe the recording after uploading it." The recording settings screen 210 also has a "Start Recording Now" button 217. When the user checks checkbox 209, the recorded video is accompanied by text data of the spoken words during the remote meeting. In this case, after the recording is finished, the information recording application 41 uploads the audio to the information processing system 50 along with a request for conversion to text data. Also, when the user presses the "Start Recording Now" button 217, the recording screen 220 shown in Figure 19 is displayed.

[0178] Figure 19 shows an example of the recording screen 220 displayed by the information recording application 41 during recording. Note that the explanation of Figure 19 mainly focuses on the differences from Figure 17. The recording screen 220 displays the combined image video being recorded in real time, based on the conditions set by the user in the recording settings screen 210. In Figure 19, the recording screen 220 is shown with the camera toggle button 211 on and the PC screen toggle button 212 off, displaying the panoramic image 203 and speaker image 204 (both videos) created by the meeting device 60. The recording screen 220 also displays the recording icon 225, the pause button 226, and the recording stop button 227.

[0179] The pause button 226 is used to stop recording, and after stopping, it also accepts requests to resume recording. The recording end button 227 is used to end recording. The recording ID does not change when the pause button 226 is pressed, but it does change when the recording end button 227 is pressed. After pausing, the user can also set the recording conditions that were set on the recording settings screen 210 again when resuming recording. In that case, the information recording application 41 may create multiple recording files each time recording is stopped (for example, when the recording end button 227 is pressed), or it may combine multiple files to form a single continuous video (for example, when the pause button 226 is pressed). Also, when the information recording application 41 plays a combined video, it may play multiple recording files continuously as a single video.

[0180] The recording screen 220 also has a button 221 for retrieving information from the calendar, a meeting name field 222, a time field 223, and a location field 224. The button 221 for retrieving information from the calendar is used by the user to retrieve meeting information from the meeting management system 9. When the button 221 is pressed, the information recording application 41 retrieves and displays a list of meetings that the user has permission to view from the information processing system 50. The user selects the remote meeting to be held from the meeting list. As a result, the meeting information is reflected in the meeting name field 222, the time field 223, and the location field 224. The meeting name field 222 displays the title of the meeting information, the time field 223 displays the start and end times, and the location field 224 displays the location. In addition, the meeting information and recording information in the meeting management system are associated with each other by the meeting ID.

[0181] Once the remote meeting ends and the user stops recording, a combined video with audio is created.

[0182] Figure 20 shows an example of the meeting list screen 230 displayed by the information recording application 41. The meeting list screen 230 is a list of meetings, and can display a list of recorded information from remote meetings. It also includes meetings held only within a specific meeting room, regardless of whether they were remote meetings. The meeting list screen 230 displays meeting information that the logged-in user has permission to view in the meeting information storage unit 5001. Information from video recordings stored in the information storage unit 1001 may be further integrated.

[0183] The meeting list screen 230 is displayed when the user selects the meeting list tab 231 on the initial screen 200 in Figure 17. The meeting list screen 230 displays a list 236 of the recorded information that this user has permission to view. The meeting creator (minutes creator) can set viewing permissions for participants. The meeting list may be a list of saved recorded information, or a list of meeting schedules or meeting data.

[0184] The meeting list screen 230 has the following items: checkbox 232, update date and time 233, title 234, and status 235.

[0185] Checkbox 232 accepts the selection of recording files. Checkbox 232 is used when the user wants to delete multiple recording files at once.

[0186] • The update time 233 indicates the start and end times of the combined image video recording. If it has been edited, the edit time can be used instead.

[0187] • The title is 234, which is the title of the meeting (agenda, etc.). It can be copied from the meeting information or set by the user.

[0188] Status 235 indicates whether the combined image / video has been uploaded to the information processing system 50. If it has not been uploaded, "Local PC" is displayed; if it has been uploaded, "Uploaded" is displayed. If it has not been uploaded, the upload button is displayed. If there are unuploaded combined image / videos, the information recording application 41 should automatically upload them when the user logs into the information processing system 50.

[0189] When the user selects a title or other item from the list of combined images and videos 236 using the pointing device, the information recording application 41 displays a recording playback screen, but this is omitted in this embodiment. On the recording playback screen, it is possible to play back the combined images and videos.

[0190] Ideally, users should be able to filter meetings by update date, title, keywords, etc. Furthermore, if the number of displayed meetings is large and it is difficult to find the relevant one, a search function should be available allowing users to input terms to narrow down the recorded information based on keywords found in meeting statements or titles. This search function would enable users to quickly find the desired recorded information, even when dealing with a large volume of records. Additionally, the meeting list screen 230 may allow users to sort meetings by update date or title.

[0191] <Action or process> First, referring to Figure 21, we will explain the overall flow of how the meeting device 60 starts and ends the extraction of the display device image 215. Figure 21 is a sequence diagram illustrating how the meeting device 60 starts and ends the extraction of the display device image 215 in response to the start and end of operation on the electronic whiteboard 2.

[0192] S1: A user participating in the meeting operates the electronic whiteboard 2. For example, the user can write by hand with the electronic pen 490 or select menus.

[0193] S2: The contact position detection unit 31 of the electronic whiteboard 2 detects the contact position of the electronic pen 490, and the operation detection unit 38 detects the start of operation. The user detection unit 39 may also detect the approach of a user.

[0194] S3: The communication unit 36 ​​of the electronic whiteboard 2 transmits the start of operation to the terminal device 10. The method by which the electronic whiteboard 2 and the terminal device 10 communicate will be described later (Figures 22 to 25).

[0195] S4: The communication unit 11 of the information recording application 41 receives the operation start notification, so the extraction request unit 23 sends an electronic whiteboard recording start request (extraction request for display device image 215) to the meeting device 60 via the device communication unit 16.

[0196] S5: The terminal communication unit 61 of the meeting device 60 receives a cropping request, and the display device image generation unit 66 generates a display device image 215 from the panoramic image. The method for detecting the direction of the electronic whiteboard 2 will be described later. From here on, the meeting device 60 transmits the display device image 215 to the terminal device 10 until it receives a request to end the electronic whiteboard recording. In this case, the meeting device 60 may reduce the number of speaker images transmitted to the terminal device 10, or it may keep the same number.

[0197] S6: Users participating in the meeting end their operation of the electronic whiteboard 2. Operation can be terminated in two ways: active termination, such as turning off the power or pressing the termination button, or passive termination, such as inactivity for a certain period of time.

[0198] S7: The operation detection unit 38 of the electronic whiteboard 2 detects the end of the operation. The user detection unit 39 may also detect that the user has moved away.

[0199] S8: The communication unit 36 ​​of the electronic whiteboard 2 transmits a message to the terminal device 10 indicating the end of the operation.

[0200] S9: The communication unit 11 of the information recording application 41 receives the operation start notification, so the extraction request unit 23 sends an electronic whiteboard recording termination request (a request to terminate the extraction of the display device image 215) to the meeting device 60 via the device communication unit 16.

[0201] S10: The terminal communication unit 61 of the meeting device 60 receives a notification that the cropping is complete, and the display device image generation unit 66 terminates generating the display device image 215 from the panoramic image.

[0202] As described above, the meeting device 60 switches between starting and ending the transmission of the display device image 215.

[0203] S11: The terminal communication unit 61 of the meeting device 60 transmits a panoramic image, a speaker image, a display device image 215 (if not present), and audio data (synthesized audio data) to the terminal device 10.

[0204] S12: The device communication unit 16 of the information recording application 41 receives these, and the recording control unit 17 records the panoramic image, speaker image, display device image 215 (if not present), and audio data as recording information. Upon completion of recording, the recording control unit 17 sends the combined image video (with audio data) to the storage service system 70, and the audio data processing unit 18 requests the information processing system 50 to convert the audio data into text data, thereby sending the text data to the storage service system 70. Preferably, the combined image video and text data are associated with the conference ID and saved to the same URL, etc.

[0205] <An example of the process involved in communication between an electronic whiteboard and a terminal device> First, the user registers a new meeting with the information processing system 50 using the meeting registration screen 250, as shown in Figure 22. Figure 22 is an example of the meeting registration screen 250 displayed by the terminal device 10.

[0206] Item 241, "Meeting Name," is the title of the meeting.

[0207] Item 242, "Organizer," contains information about the person who organized the meeting, and is usually the person who operates the meeting registration screen.

[0208] The date and time field 243 is for entering the start and end dates and times of the meeting. The date and time field 243 can be selected from a calendar.

[0209] Participant item 244 is a participant in the meeting and can be selected from a list of users registered in the account information storage unit 5004.

[0210] Item 245 for the electronic whiteboard is a selection field for the electronic whiteboard 2 to be used in the meeting, and in the account information storage unit 5004, it is possible to select from the list of accounts whose type is electronic whiteboard 2.

[0211] The meeting device item 246 is a selection field for the meeting device to be used in the meeting, and the type can be selected from the list of meeting devices in the account information storage unit 5004.

[0212] As shown in Figure 22, when creating a new meeting, the user can select the electronic whiteboard 2 and meeting device to be used in that meeting.

[0213] Next, as shown in Figure 23, the user registers the meeting device 60 and the electronic whiteboard 2 with the information processing system 50. Since the meeting has already been created, the user only needs to select the meeting. Figure 23 is an example of a sequence diagram illustrating the process by which the user registers the meeting device 60 and the electronic whiteboard 2 with the information processing system 50.

[0214] S21: As shown in Figure 22, the user creates the meeting information.

[0215] S22: The operation reception unit 12 of the terminal device 10 receives the conference information, and the communication unit 11 transmits it to the information processing system 50. The communication unit 51 of the information processing system 50 receives the conference information, and the communication management unit 54 registers it in the conference information storage unit 5001.

[0216] S23: When a user starts a meeting, they may move to the meeting room and connect the meeting device 60 to the terminal device 10. The meeting device 60 starts up when the USB cable is connected, and communication with the meeting device 60 (transmission of various images) begins when the information recording application 41 is launched.

[0217] S24: The user selects the meeting to be held from the meeting list screen 230 displayed by the information recording app 41.

[0218] S25: The operation reception unit 12 of the information recording application 41 receives the operation, and the communication unit 11 requests registration to the meeting by specifying the equipment identification information and conference ID of the meeting device 60. The communication unit 51 of the information processing system 50 receives the request to register to the meeting and determines that it is a meeting device 60 that is registered in the meeting information. The equipment management unit 55 associates the equipment identification information of the meeting device 60 with the conference ID.

[0219] S26: Next, the user selects the meeting to be held from the meeting list screen 230 displayed on the electronic whiteboard 2.

[0220] S27: The communication unit 36 ​​of the electronic whiteboard 2 requests registration for the meeting, specifying the device identification information and meeting ID of the electronic whiteboard 2. The communication unit 51 of the information processing system 50 receives the registration request for the meeting and determines that it is the electronic whiteboard 2 registered in the meeting information. The device management unit 55 associates the device identification information of the electronic whiteboard 2 with the meeting ID.

[0221] In this way, the information processing system 50 associates the meeting device 60 (terminal device 10) with the electronic whiteboard 2, thereby enabling the terminal device 10 and the electronic whiteboard 2 to communicate with each other via the information processing system 50. When the meeting device 60 (terminal device 10) and the electronic whiteboard 2 transmit their respective identification information to the information processing system 50, the information processing system 50 can identify the other device registered with the same conference ID using the association information.

[0222] In Figures 22 and 23, the user has registered meeting information in advance, but even without registering meeting information beforehand, the user can associate the meeting device 60 and the electronic whiteboard 2 and register them with the information processing system 50 at the start of the meeting.

[0223] Figure 24 is an example of a sequence diagram illustrating the process by which a user registers a meeting device 60 and an electronic whiteboard 2 with the information processing system 50, even though no meeting information has been created yet.

[0224] S31: The user inputs an operation to start registering the association with the meeting device 60 on the electronic whiteboard 2 in the conference room.

[0225] S32: The electronic whiteboard 2 communicates with the pre-configured information processing system 50 in response to the operation, and registers the device identification information and the fact that it can be associated with the meeting.

[0226] S33: Furthermore, the code generation unit 35 of the electronic whiteboard 2, which is placed in the conference room used for meetings, generates a two-dimensional code encoded with its own device identification information and a statement indicating that it is device identification information, and the display unit 34 displays it. The electronic whiteboard 2 may also include a password for authenticating other devices.

[0227] S34: The user enters the conference room where the electronic whiteboard 2 is installed, bringing the terminal device 10 and the meeting device 60, and connects the terminal device 10 and the meeting device 60 with a USB cable. The meeting device 60 starts up due to power supply from the USB cable or when the power is turned on. This puts the meeting device 60 into standby mode. The user also starts the information recording application 41 on the terminal device 10. When the information recording application 41 starts communicating with the meeting device 60, the meeting device 60 starts capturing images and collecting sound. The panoramic image generation unit 62 of the meeting device 60 generates a panoramic image of the surroundings, including a two-dimensional code.

[0228] S35: The terminal communication unit 61 of the meeting device 60 transmits the panoramic image and the device identification information of the meeting device 60 to the terminal device 10.

[0229] S36: The device communication unit 16 of the terminal device 10 receives the panoramic image and the device identification information of the meeting device 60, and the analysis unit 22 detects the two-dimensional code displayed by the electronic whiteboard 2 from the panoramic image. The analysis unit 22 decodes the two-dimensional code and, if it determines that it contains information indicating that the device is usable for the meeting, it obtains the device identification information of the electronic whiteboard 2 from the two-dimensional code. The communication unit 11 of the information recording application 41 sends a registration request for the meeting to the information processing system 50, specifying the device identification information of the electronic whiteboard 2 and the device identification information of the meeting device 60. Note that the two-dimensional code can also be decoded by the meeting device 60.

[0230] S37, S38: When the communication unit 51 of the information processing system 50 receives a registration request for a meeting (device identification information, device identification information of the meeting device 60), the communication management unit 54 issues a meeting ID. The device management unit 55 then associates the device identification information of the electronic whiteboard 2, the device identification information of the meeting device 60, and the meeting ID, and stores them in the association information storage unit 5003.

[0231] The communication unit 51 of the information processing system 50 notifies the terminal device 10 and the electronic whiteboard 2 of the conference ID and the completion of registration for the conference on the electronic whiteboard 2. The communication unit 11 of the terminal device 10 receives and stores the conference ID. Similarly, when the communication unit 36 ​​of the electronic whiteboard 2 receives the conference ID and password, it verifies the password and stores the conference ID if they match. The terminal device 10 receives the conference ID as a response to the communication in step S36. The electronic whiteboard 2 receives the conference ID and password as a response to polling (requesting the mapping of device identification information to the conference) to the information processing system 50. The electronic whiteboard 2 and the information processing system 50 may communicate using a bidirectional communication method such as WebSocket, which enables push-type communication from the information processing system 50 to the electronic whiteboard 2.

[0232] From this point forward, terminal device 10 and electronic whiteboard 2 will attach the conference ID (or identification information for each device) to the data they transmit.

[0233] In this way, the information processing system 50 associates the meeting device 60 (terminal device 10) with the electronic whiteboard 2, thereby enabling the terminal device 10 and the electronic whiteboard 2 to communicate with each other via the information processing system 50.

[0234] In Figures 21 to 24, the terminal device 10 and the electronic whiteboard 2 can communicate via the information processing system 50. However, if the terminal device 10 and the electronic whiteboard 2 are connected to the same network, they can communicate using IP addresses. The same network refers to cases where the network address is the same or the destination SSID (Service Set Identifier) ​​is the same.

[0235] Figure 25 is an example of a sequence diagram illustrating the communication method between terminal device 10 and electronic whiteboard 2 when terminal device 10 and electronic whiteboard 2 are connected to the same network.

[0236] S41: The user inputs an operation to start associating with the meeting device 60 on the electronic whiteboard 2 in the conference room.

[0237] S42: The electronic whiteboard 2 accepts the command, and the display unit 34 displays the IP address and password of the electronic whiteboard 2.

[0238] S43: The user sees the displayed IP address and password and enters them into the information recording application 41 of the terminal device 10. The operation reception unit 12 of the terminal device 10 accepts the input.

[0239] S44: The communication unit 11 of the terminal device 10 connects to the electronic whiteboard 2 using an IP address and sends a password. Upon successful password verification, the terminal device 10 and the electronic whiteboard 2 become able to communicate.

[0240] <Determining the orientation of the electronic whiteboard in a panoramic image> Next, we will explain how to determine the orientation of electronic whiteboard 2 in the panoramic image. There are mainly four methods for determining the orientation. 1. The user selects a panoramic image at the start of the meeting. 2. The electronic whiteboard 2 displays a specific image (such as a QR code), and the terminal device 10 or the meeting device 60 recognizes it from the panoramic image captured by the imaging unit 601 of the meeting device 60. 3. The electronic whiteboard 2 outputs a specific sound, which the meeting device 60 recognizes with the microphone 608. 4. The shape of the electronic whiteboard 2 is learned by an arbitrary information processing device using machine learning, and the terminal device 10 or the meeting device 60 recognizes it from the panoramic image captured by the camera (imaging unit 601) of the meeting device 60.

[0241] <<1. The user selects a panoramic image at the start of the meeting.>> Figure 26 illustrates the operation method for setting the orientation of the electronic whiteboard 2 by pressing the position registration button 207. When the position registration button 207 is pressed, a panoramic image 203 is displayed in a pop-up window. For example, the user moves a rectangular window 206 on the panoramic image 203 using a pointing device such as a mouse or touch panel. The user aligns the window 206 with the electronic whiteboard 2 or podium included in the panoramic image 203.

[0242] Figure 27 shows a screen for confirming the direction set by the user. When the user presses the OK button 208, the direction of the electronic whiteboard 2 relative to the panoramic image is set. The direction set by the user is transmitted to the meeting device 60, and the display device identification unit 67 of the meeting device 60 stores it.

[0243] <<2. The interactive whiteboard displays a specific image (such as a QR code), and the terminal device or meeting device recognizes it from the panoramic image captured by the meeting device's imaging unit. 3. The interactive whiteboard outputs a specific sound, which the meeting device recognizes via its microphone.>> Figure 28 is an example of a sequence diagram illustrating the process by which the information recording application 41 determines the orientation of the electronic whiteboard 2 based on specific video or audio.

[0244] S51: The user inputs an operation to determine the direction of the electronic whiteboard 2 into the information recording application 41 of the terminal device 10. The operation reception unit 12 receives the operation.

[0245] S52: The communication unit 11 of the information recording application 41 requests the output of a specific video or a specific sound from the electronic whiteboard 2. The communication unit 11 may communicate with the electronic whiteboard 2 via the LAN or via the information processing system 50.

[0246] S53: The communication unit 36 ​​of the electronic whiteboard 2 receives a request, and the code generation unit 35 generates a two-dimensional code as a specific image. The display unit 34 displays the two-dimensional code on the display 480. As will be described later, this two-dimensional code includes information indicating the direction of the electronic whiteboard 2 and whether it is on the right or left side. The information indicating the right or left side is not necessarily included.

[0247] Furthermore, the communication unit 36 ​​of the electronic whiteboard 2 receives a request, and the voice data generation unit 27 outputs a sound of a specific frequency from the speaker 450. Either the code generation unit 35 or the voice data generation unit 27 needs to be operational, but both may be operational.

[0248] S54: The panoramic image generation unit 62 of the meeting device 60 repeatedly generates panoramic images, so if a two-dimensional code is within the field of view, it automatically generates a panoramic image including the two-dimensional code. Also, the sound collection unit 64 of the meeting device 60 repeatedly collects sound, so it automatically collects sound of a specific frequency. The display device identification unit 67 obtains a frequency spectrum by performing a Fourier transform on the audio data and identifies two directions from which sound of a predetermined frequency and above a threshold is arriving. The display device identification unit 67 of the electronic whiteboard 2 stores the direction of the electronic whiteboard 2 (A° to B° within the 360° horizontal direction). It is preferable that this sound is in the ultrasonic frequency band, as it will not be audible to the user.

[0249] S55: The terminal communication unit 61 of the meeting device 60 transmits the panoramic image to the terminal device 10.

[0250] S56: The device communication unit 16 of the information recording application 41 receives a panoramic image. The analysis unit 22 detects the two-dimensional code displayed on the electronic blackboard 2 from the panoramic image. When the analysis unit 22 decodes the two-dimensional code and determines that the direction of the electronic blackboard 2 is embedded, the analysis unit 22 identifies the positions of the left and right ends of the two-dimensional code in the horizontal direction in the panoramic image. Details will be described in FIG. 29. This process can also be performed by the display device identification unit 67 of the meeting device 60.

[0251] S57: The device communication unit 16 of the terminal device 10 transmits the direction of the electronic blackboard 2 (A° to B° out of 360° in the horizontal direction) to the meeting device 60. The terminal communication unit 61 of the meeting device 60 receives the direction of the electronic blackboard 2, and the display device identification unit 67 stores it.

[0252] S58: Since the direction of the electronic blackboard 2 is determined, the communication unit 11 of the information recording application 41 requests the electronic blackboard 2 to stop the output of a specific video or a specific sound.

[0253] S59: The communication unit 36 of the electronic blackboard 2 receives the request, and the display unit 34 ends the display of the two-dimensional code. The audio data generation unit 27 stops the output of a sound of a specific frequency.

[0254] <<Determination of Direction by Two-Dimensional Code>> FIG. 29 shows an example of the two-dimensional code 301 displayed by the electronic blackboard 2 as a specific video. In FIG. 29, the electronic blackboard 2 is shown in the panoramic image, and the electronic blackboard 2 displays two two-dimensional codes 301. The size (width) of the two-dimensional code 301 is set to the size obtained by dividing the width of the display of the electronic blackboard 2 into three parts. Also, two two-dimensional codes 301 are displayed. One is displayed靠右 on the display 480, and the other is displayed靠左. Each two-dimensional code 301 contains information on whether it is displayed on the right or the left. In this embodiment, right and left are determined in the state facing the electronic blackboard 2 from the user.

[0255] Figure 30 illustrates the method for determining direction based on the two-dimensional codes 301. The analysis unit 22 detects the two two-dimensional codes 301, thereby identifying the horizontal position information of each two-dimensional code 301. The analysis unit 22 determines that the position 253, which is moved to the left by the same amount as the width of the two-dimensional code 301 from the left end 251 of the left two-dimensional code 301, is the left edge position of the electronic whiteboard 2. The analysis unit 22 determines that the position 254, which is moved to the right by the same amount as the width of the two-dimensional code 301 from the right end 252 of the right two-dimensional code 301, is the right edge position of the electronic whiteboard 2. The analysis unit 22 determines the direction of the electronic whiteboard 2 to be from the left end position 253 to the right end position 254.

[0256] <<Direction determination by sound>> Figure 31 illustrates how the direction of the electronic whiteboard 2 is determined when the electronic whiteboard 2 emits a specific sound. As shown in Figure 31(a), speakers 450 are installed at the left and right ends of the electronic whiteboard 2. Speakers 450 may also be built into the left and right ends.

[0257] Then, as shown in Figure 31(b), the audio data generation unit 27 outputs sound from each speaker 450. The microphone 608 of the meeting device 60 is directional and identifies the direction from which the sound emitted from each speaker 450 is coming relative to the meeting device 60. Once the direction relative to the meeting device 60 is identified, the direction in the panoramic image 203 is also identified.

[0258] Furthermore, even if there is directionality, it is difficult for the display device identification unit 67 to determine the center of the speaker 450. Therefore, based on the intensity of sound at a specific frequency, the display device identification unit 67 determines the position of the speaker 450 within a certain range 261 in the left-right direction, centered on the direction with the highest intensity.

[0259] As shown in Figure 31(c), the display device identification unit 67 determines the direction of the electronic whiteboard 2 to be from the left end 262 to the right end 263 of the two ranges 261.

[0260] <<4. An arbitrary information processing device learns the shape of the electronic whiteboard using machine learning, and the terminal device or meeting device recognizes it from the panoramic image captured by the meeting device's imaging unit.>> Figure 32 illustrates the direction in which the orientation of the electronic whiteboard 2 is determined based on the electronic whiteboard 2 detected by image processing such as machine learning. First, the display device recognition unit 24 detects the shape (circumscribed rectangle) of the electronic whiteboard 2 from the panoramic image using machine learning. When the display device recognition unit 24 detects the electronic whiteboard 2 from the panoramic image, it divides the width of the electronic whiteboard 2 into three parts and determines the orientation of the electronic whiteboard 2 from a position 271, which is one-third of the width moved from the left edge of the electronic whiteboard 2, to a position 272, which is one-third of the width moved from the right edge. One-third is just one example.

[0261] The terminal device 10 transmits the orientation of the electronic whiteboard 2 to the meeting device 60, and the display device identification unit 67 saves the orientation of the electronic whiteboard 2. Note that position detection by image processing may be performed by the meeting device 60.

[0262] <State transitions when starting and ending operation of the interactive whiteboard> Next, referring to Figures 33 to 39, we will explain the state transitions at the start and end of operation for the electronic whiteboard 2.

[0263] There are two ways to detect when a user starts or finishes an operation: 1. Menu operation and writing on interactive whiteboard 2. 2. Detection of a user approaching electronic whiteboard 2

[0264] <<Menu operations, state transitions via writing>> First, referring to Figures 33 and 34, we will explain "1. Menu operation and writing on the electronic whiteboard 2".

[0265] Figure 33 is a state transition diagram based on menu operations and writing on the electronic whiteboard 2. The initial state after the start of the meeting is the operation completion state.

[0266] a. The operation end state transitions to the operation start state when a menu operation or writing is detected.

[0267] b. The operation start state transitions to the operation end state when a menu operation or writing ends.

[0268] Figure 34 is a state transition diagram that is the same in terms of menu operations and writing, but also determines on which side (left or right) of the display there is a menu operation or writing.

[0269] a. The operation end state transitions to the operation start state only on the right side when there is a menu operation or writing to the right side.

[0270] b. The operation start state only on the right side transitions to the operation start state on both sides when there is a menu operation or writing to the left side.

[0271] c. The operation end state transitions to the operation start state only on the left side when there is a menu operation or writing to the left side.

[0272] d. The operation start state only on the left side transitions to the operation start state on both sides when there is a menu operation or writing to the right side.

[0273] e. The operation start state only on the right side transitions to the operation end state when a menu operation or writing to the right side ends.

[0274] f. The operation start state on both sides transitions to the operation start state only on the right side when a menu operation or writing to the left side ends.

[0275] g. The operation start state only on the left side transitions to the operation end state when a menu operation or writing to the left side ends.

[0276] h. The operation start state on both sides transitions to the operation start state only on the left side when a menu operation or writing to the right side ends.

[0277] In this way, by dividing the display into left and right sections and determining whether there is writing on each side, the meeting device 60 can isolate only the right or left side of the electronic whiteboard 2.

[0278] Figure 35 is a flowchart illustrating the process by which the electronic whiteboard 2 determines the transition to the operation start state. The electronic whiteboard 2 and the meeting device 60 are registered in association with the information processing system 50, and the meeting starts when the recording start button 217 is pressed (S101). Note that the so-called remote meeting starts when the remote meeting application 42 connects with another location 101. Step S101 signifies the start of recording.

[0279] The operation reception unit 12 determines whether or not the recording end button has been pressed (S102). The information recording application 41 considers the meeting to have ended when the recording ends. When the meeting ends (Yes in S102), the process in Figure 35 also ends.

[0280] Until the meeting ends (No. S102), the operation detection unit 38 of the electronic whiteboard 2 determines whether a menu operation or writing operation has been detected based on the contact of a pen or fingers detected by the contact position detection unit 31 (S103). If no menu operation or writing operation is detected (No. S103), the process returns to S102.

[0281] If a menu operation or writing operation is detected (Yes in S103), the operation detection unit 38 determines whether the writing was done on the left or right side of the display of the electronic whiteboard 2 based on the contact position (S104).

[0282] The operation detection unit 38 determines whether the operation has started (S105). If the operation has already started (Yes in S105), the process proceeds to step S102.

[0283] If the operation is not currently in the operation start state (No. in S105), the operation detection unit 38 transitions to the operation start state (S106).

[0284] The communication unit 36 ​​transmits an operation start signal to the terminal device 10 (S107). This allows the terminal device 10 to send a request to the meeting device 60 to start recording the electronic whiteboard.

[0285] Figure 36 is a flowchart illustrating the process by which the recording information creation system 100 determines the transition to the operation completion state.

[0286] The electronic whiteboard 2 and the meeting device 60 are registered in association with the information processing system 50, and the meeting starts when recording is initiated (S201).

[0287] The operation reception unit 12 determines whether or not the recording end button has been pressed (S202). The information recording application 41 considers the meeting to have ended when the recording ends. When the meeting ends (Yes in S202), the process in Figure 36 also ends.

[0288] Until the meeting ends (No. S202), the operation detection unit 38 of the electronic whiteboard 2 determines whether a menu operation or writing operation has been detected based on the contact of a pen or fingers detected by the contact position detection unit (S203).

[0289] If no menu operation or write operation is detected (No in S203), the operation detection unit 38 determines whether the operation is currently in the start state (S204). If a menu operation or write operation is detected (Yes in S203), the process shown in Figure 35 is executed.

[0290] If the operation is currently in the start state (Yes in S204), the operation detection unit 38 determines whether the operation termination condition is met (S205). The operation termination condition may be power off, or no contact with the touch panel is detected for a certain period of time or longer. If the operation termination condition is not met (No in S205), the process returns to step S202.

[0291] If the operation termination condition is met (Yes in S205), the operation detection unit 38 transitions to the operation termination state (S206).

[0292] The communication unit 36 ​​transmits an operation completion message to the terminal device 10 (S207). This allows the terminal device 10 to send a request to the meeting device 60 to end the electronic whiteboard recording.

[0293] <<State transitions due to user approach and distance>> Next, referring to Figure 37, we will explain "2. Detection of a user approaching the electronic whiteboard 2".

[0294] Figure 37 is a state transition diagram based on the user's approach to the electronic whiteboard 2. a. The initial state is set to the operation completion state. When the user detection unit 39 detects a user approaching the electronic whiteboard 2, the state transitions to the user approaching state. b. When the user detection unit 39 detects that the user has stopped while the user is approaching, the system transitions to the operation start state. If the user detection unit 39 detects that the user has moved away while the user is approaching or the operation has started, the system transitions to the operation completion state.

[0295] By creating a user approaching state in this way, it is possible to avoid interpreting a user simply walking past the interactive whiteboard 2 as an indication that the user has started operating the interactive whiteboard 2. Furthermore, it is possible to detect the start of user operation more quickly compared to detecting operation on the interactive whiteboard 2.

[0296] Figure 38 is a flowchart illustrating the process by which the recording information creation system 100 determines the transition to the operation start state. Steps S301 and S302 can be the same as in Figure 35.

[0297] The user detection unit 39 of the electronic whiteboard 2 determines whether or not a user is approaching (S303). If no user is approaching, the process proceeds to step S302.

[0298] If a user's approach is detected (Yes in S303), the user detection unit 39 determines whether the system is currently in an operation start state (S304). If the system is already in an operation start state (Yes in S304), the process returns to step S302.

[0299] If the operation is not currently in the start state (No. in S304), the user detection unit 39 determines whether or not a user is currently approaching (S305).

[0300] If the user is currently approaching (Yes in S305), the user detection unit 39 transitions to the operation start state (S306).

[0301] If the user is not currently in the approaching user state (No. in S305), the user detection unit 39 transitions to the approaching user state (S307).

[0302] The communication unit 36 ​​transmits an operation start signal to the terminal device 10 (S308). This allows the terminal device 10 to send a request to the meeting device 60 to start recording the electronic whiteboard.

[0303] Figure 39 is a flowchart illustrating the process by which the recording information creation system 100 determines the transition to the operation completion state. Steps S401 and S402 can be the same as in Figure 36.

[0304] The user detection unit 39 of the electronic whiteboard 2 determines whether or not the user has moved away (S403). If the user has not moved away (No. in S403), the process proceeds to step S402.

[0305] If the user is detected to be moving away (Yes in S403), the user detection unit 39 determines whether the system is currently in the operation start state or the user is approaching state (S404). If the system is not in the operation start state or the user is approaching state (No in S404), the process proceeds to step S402.

[0306] If the system is currently in the operation start state or the user approach state (Yes in S404), the user detection unit 39 transitions to the operation end state (S405).

[0307] The communication unit 36 ​​transmits an operation completion message to the terminal device 10 (S406). This allows the terminal device 10 to send a request to the meeting device 60 to end the electronic whiteboard recording.

[0308] <Example of cropping an electronic whiteboard image from a panoramic photo> Upon receiving a request from the terminal device 10 to start recording the electronic whiteboard, the meeting device 60 begins the process of extracting a display image 215 of a specific area in the predetermined direction of the electronic whiteboard 2 from the panoramic image. The specific area is an area that includes at least a part of the electronic whiteboard 2. Upon receiving a request from the terminal device 10 to end recording the electronic whiteboard, the meeting device 60 terminates the extraction of the display image 215 from the panoramic image.

[0309] Furthermore, if the meeting device 60 receives notification that an operation has been performed on either the left or right side of the electronic whiteboard 2, it performs a process to extract only the display image 215 (hereinafter referred to as the half-display image 216) of the right or left half where the operation occurred. In this case, the specific area refers to half of the electronic whiteboard 2. The half-display image 216 is an area obtained by dividing the direction of the electronic whiteboard 2 equally to the left and right, as determined in advance. It does not necessarily have to be equally divided to the left and right, and may be divided to include all coordinates where the pen made contact within a certain period of time. If an operation has been performed on either the left or right side, it is possible that there are multiple users using the electronic whiteboard 2, so the meeting device 60 performs a process to extract the left and right sides of the electronic whiteboard 2.

[0310] First, Figure 40 shows an example of a predetermined orientation of the electronic whiteboard 2 in a panoramic image. Since the meeting device 60 can capture images in the horizontal direction of 360° with the reference direction being 0°, the orientation of the electronic whiteboard 2 is predetermined as A°~B°. In this case, the specific area refers to the region from A° to B° that includes the entire electronic whiteboard 2.

[0311] Figure 41 shows a display device image 215 cropped based on the orientation of the electronic whiteboard 2. Figure 41 shows the case where the electronic whiteboard recording start request is sent to the meeting device 60 without distinguishing between the left and right sides of the electronic whiteboard 2. In this case, the display device image generation unit 66 crops the image as is, within the range of the orientation of the electronic whiteboard 2.

[0312] Figure 42 shows a half-display image 216 when a request to start recording the electronic whiteboard 2, specifying either the left or right side, is sent to the meeting device 60. Figure 42(a) shows the half-display image 216 when a request to start recording the electronic whiteboard, specifying the right side, is sent to the meeting device 60. Figure 42(b) shows the half-display image 216 when a request to start recording the electronic whiteboard, specifying the left side, is sent to the meeting device 60.

[0313] Figure 43 shows an example of speaker image 204 display by the display control unit 13 of the information recording application 41. In Figure 43, two speaker images 204 of the speaker who spoke are displayed, regardless of the electronic whiteboard 2.

[0314] Figure 44 shows the display control unit 13 of the information recording application 41 displaying a half-display image 216 when a request to start recording the electronic whiteboard, specified on the right, is sent to the meeting device 60. The half-display image 216 is displayed at the same size as the speaker image 204.

[0315] As shown in Figure 45, the display control unit 13 of the information recording application 41 may display the user facing the electronic whiteboard 2 larger than the speaker image 204. Figure 45 shows the display device image 215 displayed by the information recording application 41 when an electronic whiteboard recording start request without left / right specification is sent to the meeting device 60. The display device image 215 is displayed larger than the speaker image 204. This makes it easier for viewers to focus on important information handwritten on the electronic whiteboard 2.

[0316] In Figure 45, the display device image 215 is shown, but the display control unit 13 of the information recording application 41 can similarly display the half-display device image 216 larger than the speaker image.

[0317] In addition, the user operating the electronic whiteboard 2 may also speak. In this case, if both the speaker image 204 and the display device image 215 are extracted, the speaker and operator will be extracted overlappingly, so it is preferable to omit the extraction of the speaker image 204. In this case, the speaker image generation unit 63 acquires the direction of the electronic whiteboard 2, and when an electronic whiteboard recording start request is transmitted, if the direction of the speaker and the direction of the electronic whiteboard 2 overlap by a certain amount, it does not generate a speaker image. If the user is using the electronic whiteboard 2 and the user is the speaker, the display device image generation unit 66 can generate a display device image 215 that includes the speaker and the electronic whiteboard 2.

[0318] Furthermore, as shown in Figure 46, the user may have the display control unit 13 of the information recording application 41 prioritize the display of the display device image 215. Figure 46 shows the display device image 215 that is prioritized by user operation. Figure 46 is the same screen as the recording screen 220 in Figure 19. The recording screen 240 in Figure 46 has a checkbox 218 associated with the text "Prioritize display of electronic whiteboard" and a priority switching button 219.

[0319] When the user checks checkbox 218 or presses the priority switching button 219, the display control unit 13 of the information recording application 41 enlarges the display device image 215. For example, the display control unit 13 displays the display device image 215 at a larger size than the speaker image 204. The user can enlarge the display device image 215 if they want to check the content displayed on the electronic whiteboard 2. The display control unit 13 may also enlarge only the speaker image 204 independently. The information recording application 41 can similarly enlarge the half-display device image 216.

[0320] <Main effects> As described above, the recording information creation system 100 of this embodiment extracts the display device image 215 when the electronic whiteboard 2 is operated, so that when new handwriting is done on the electronic whiteboard 2, the handwritten data can be displayed. When the handwriting is finished, the display of the display device image 215 can also be ended. This makes it possible to transmit the state of the electronic whiteboard 2 and its surroundings while the user is using the electronic whiteboard 2 to a remote location.

[0321] <Other application examples> Although the best mode for carrying out the present invention has been described above using examples, the present invention is not limited in any way to these examples, and various modifications and substitutions can be made without departing from the spirit of the present invention.

[0322] For example, the terminal device 10 and the meeting device 60 may be integrated into one unit. Alternatively, the meeting device 60 may be attached externally to the terminal device 10. Furthermore, the meeting device 60 may consist of a 360-degree camera, a microphone, and a speaker connected by cables.

[0323] Meeting devices 60 may also be placed at other locations 101. These other locations 101 will separately use meeting devices 60 to create combined image / video and text data. Furthermore, multiple meeting devices 60 may be placed at a single location. In this case, multiple recordings will be created for each meeting device 60.

[0324] Furthermore, the arrangement of the panoramic image 203, speaker image 204, and application screen in the combined image video used in this embodiment is merely an example. The panoramic image 203 may be at the bottom and the speaker image 204 at the top, or the user may change the arrangement, or individually switch the display of the panoramic image 203 and speaker image 204 during playback.

[0325] Furthermore, the configuration examples shown in Figure 9 and other figures are divided according to their main functions to facilitate understanding of the processing performed by the terminal device 10, meeting device 60, and information processing system 50. The present invention is not limited by the way the processing units are divided or the names of those units. The processing of the terminal device 10, meeting device 60, and information processing system 50 can be further divided into more processing units depending on the processing content. Alternatively, a single processing unit can be divided to include even more processing.

[0326] Furthermore, the apparatus described in the examples represents only one of several computing environments for carrying out the embodiments disclosed herein. In one embodiment, the information processing system 50 includes multiple computing devices, such as a server cluster. The multiple computing devices are configured to communicate with each other via any type of communication link, including networks and shared memory, and perform the processing disclosed herein.

[0327] Furthermore, the information processing system 50 can be configured to share the disclosed processing steps, such as those shown in Figure 21, in various combinations. For example, a process executed by a predetermined unit can be executed by multiple information processing devices of the information processing system 50. Also, the information processing system 50 may be consolidated into a single server device or divided into multiple devices.

[0328] Each of the embodiments described above can be implemented by one or more processing circuits. Here, "processing circuit" as used herein includes processors programmed to execute each function by software, such as processors implemented by electronic circuits, as well as devices such as ASICs (Application Specific Integrated Circuits), DSPs (Digital Signal Processors), FPGAs (Field Programmable Gate Arrays), and conventional circuit modules designed to execute each of the functions described above. [Explanation of Symbols]

[0329] 10 Terminal devices 50 Information Processing Systems 60 Meeting Devices 100 Record Information Creation System [Prior art documents] [Patent Documents]

[0330] [Patent Document 1] Japanese Patent Publication No. 2018-032098

Claims

1. A display system having a display device, A display device identification unit that identifies the display device based on information relating to the display device, A detection unit that detects when a user is using the display device, The system includes a display control unit that displays a display device image of a specific region including at least a part of the display device from image information captured by the imaging device, The display device has a communication unit that transmits to a terminal device capable of communicating with the imaging device that the detection unit has detected that the user is using the display device. The detection unit detects the contact position of the touch panel of the display device, The communication unit transmits the contact position detected by the detection unit to the terminal device. The display control unit has a display image that is modified according to the contact position received from the display device, The system is characterized by displaying a display image that includes the right half of the touch panel when the user writes to the right half of the touch panel, a display image that includes the left half of the touch panel when the user writes to the left half of the touch panel, and a display image that includes the entire display when the user writes to both the left and right areas of the touch panel. Display system.

2. The display device and the imaging device have an information processing system that can communicate via a network. The information processing system has the display device and the imaging device registered as devices used for communication with the user. The display system according to claim 1, characterized in that the communication unit transmits to the terminal device, via the information processing system, that the detection unit has detected something.

3. The display device has a display unit that displays device identification information of the display device, The imaging device and, The display device and the imaging device are connected to an information processing system that can communicate via a network. The display unit includes an analysis unit that acquires device identification information of the display device from the image information which includes device identification information of the display device captured by the display unit, The information processing system accepts registration of the display device and the imaging device as devices used for user communication, based on the device identification information of the display device and the imaging device received from the terminal device. The display system according to claim 1, characterized in that the communication unit transmits to the terminal device, via the information processing system, that the detection unit has detected something.

4. The display system according to any one of claims 1 to 3, characterized in that the detection unit detects when the user has started using the display device by detecting contact with the touch panel of the display device or the approach of the user.

5. The detection unit detects that the use of the display device has ended, The display system according to any one of claims 1 to 4, characterized in that the display control unit terminates the display of the display device image when it receives from the display device that the use of the display device has ended.

6. The display system according to claim 5, characterized in that the detection unit detects that the use of the display device has ended when there is no contact with the touch panel of the display device for a certain period of time, or when the user moves away.

7. The display control unit is characterized in that it displays the display image of the specific region based on the range in which the display device is visible, as described in any one of claims 1 to 6.

8. The display device identification unit identifies the range in which the display device is captured by detecting a specific image displayed by the display device captured by the imaging device. The display system according to claim 7, characterized in that the display control unit displays the display image of the specific region based on the range in which the display device is visible.

9. The display device identification unit identifies the display device by detecting the direction of a specific sound output by the display device, The display control unit is characterized in that it displays the display image of the specific region based on the direction of the specific sound, as described in any one of claims 1 to 6.

10. The display device identification unit detects the display device by image processing and identifies the area in which the display device is captured. The display system according to claim 7, characterized in that the display control unit displays the display image of the specific region based on the range in which the display device is visible.

11. The display system according to claim 1, characterized in that the display control unit displays the display image of the specific region, which is modified according to the contact position received from the display device.

12. Having the aforementioned imaging device, The imaging device has a speaker image generation unit that detects the direction of the speaker and generates a speaker image from the image information captured by the imaging device. The display system according to any one of claims 1 to 11, characterized in that when the detection unit detects that a user is using the display device, and the user is the speaker, the display control unit displays a display device image of a specific region including the speaker and at least a part of the display device from the image information.

13. When the detection unit detects that the user is using the display device, and the direction of the display device and the direction of the speaker overlap by a certain amount, The display system according to claim 12, characterized in that the speaker image generation unit does not generate a speaker image of the speaker that overlaps with the orientation of the display device.

14. The display system according to claim 12 or 13, characterized in that when the display control unit receives an operation to prioritize displaying the display device image, it displays the display device image larger than the speaker image.

15. A display method performed by a display system having a display device, The steps include: the display device identification unit identifies the display device based on information relating to the display device; The detection unit detects that the user is using the display device, The display control unit performs the step of displaying a display device image of a specific region including at least a part of the display device from image information captured by the imaging device, The communication unit of the display device transmits to a terminal device that can communicate with the imaging device that the detection unit has detected that the user is using the display device. The detection unit performs the step of detecting the contact position with respect to the touch panel of the display device, The communication unit transmits the contact position detected by the detection unit to the terminal device. The display control unit has modified the display image according to the contact position received from the display device, The steps include: displaying a display image that includes the right half of the touch panel if the user writes to the right half of the touch panel; displaying a display image that includes the left half if the user writes to the left half of the touch panel; and displaying a display image that includes the entire display if the user writes to both the left and right areas of the touch panel. A display method characterized by performing the following.

16. An imaging device capable of communicating with a terminal device, A display device identification unit that identifies the display device based on information about the display device, When the terminal device receives a notification that a user is using the display device, the display device image generation unit generates a display device image of a specific region including at least a part of the display device from the image information captured by the imaging device, It includes a terminal communication unit that transmits the display device image to the terminal device, The terminal device receives from the display device that the user is using the display device and the contact position on the touch panel of the display device, The display device image generation unit, according to the contact position received from the terminal device, The system is characterized by extracting a display image including the right half when the user writes to the right half of the touch panel, extracting a display image including the left half when the user writes to the left half of the touch panel, and extracting a display image including the entire display when the user writes to both the left and right areas of the touch panel, and transmitting these images to the terminal device. Imaging device.

17. A program to be executed on a terminal device, wherein the terminal device is A communication unit that receives information that the display device has been identified based on information about the display device and that a user is using the display device, It functions as a display control unit that displays a display device image of a specific region including at least a part of the display device from image information captured by the imaging device, The communication unit receives from the display device that the user is using the display device and the contact position on the touch panel of the display device. The display control unit has a display image that is modified according to the contact position received from the display device, The system is characterized by displaying a display image that includes the right half of the touch panel when the user writes to the right half of the touch panel, a display image that includes the left half of the touch panel when the user writes to the left half of the touch panel, and a display image that includes the entire display when the user writes to both the left and right areas of the touch panel. program.

Images