Information processing equipment, information processing methods, programs, devices

Abstract

To provide a device management system that makes devices that process information related to communication available for communication. [Solution] The present invention relates to a device management system 100 having a plurality of devices and an information processing system for processing information related to communication, wherein a first device 2 has an output unit that outputs device identification information of the first device, a second device 60 that can communicate with a terminal device 10 has an acquisition unit that acquires the device identification information output by the first device and a terminal communication unit that transmits the device identification information to the terminal device, and the information processing system is characterized by having a communication unit that receives the device identification information from the terminal device and a device management unit that makes the first device available for communication in response to the receipt of the device identification information.

Description

Technical Field

[0001] The present invention relates to a device management system, an information processing system, a device management method, a program, and a device.

Background Art

[0002] There is known a remote communication system in which users at remote locations hold a meeting by transmitting images and sounds in real time from one site to one or more other sites. In remote communication, devices such as an electronic blackboard may be used.

[0003] There is known a technique for facilitating authentication of participants in a meeting (see, for example, Patent Document 1). Patent Document 1 discloses a system having imaging means for imaging the surroundings of its own device to generate video data, the imaging means reading an imaged participation certificate and analyzing the imaged participation certificate to accept participation in a meeting.

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, the conventional technology has a problem that an operation for making a device that processes information related to communication available for communication is necessary. For example, when an electronic blackboard is used in a meeting, in order to save the data handwritten on the electronic blackboard in association with the meeting, an operation for associating the electronic blackboard with the meeting by participants or the like is necessary.

[0005] In view of the above problems, an object of the present invention is to provide a device management system that makes a device that processes information related to communication available for communication.

Means for Solving the Problems

[0006] In view of the above problems, the present invention is a device management system having a plurality of devices that process information related to communication and an information processing system, The first device has an output unit that outputs device identification information of the first device, A second device capable of communicating with a terminal device includes an acquisition unit that acquires the device identification information output by the first device, and a terminal communication unit that transmits the device identification information to the terminal device. The information processing system is characterized by having a communication unit that receives the device identification information from the terminal device, and a device management unit that makes the first device available for communication in response to the receipt of the device identification information. [Effects of the Invention]

[0007] We can provide a device management system that makes available devices for processing information related to communication. [Brief explanation of the drawing]

[0008] [Figure 1] This diagram outlines the creation of recording information that saves the screen of an application running during a remote meeting, along with a panoramic image of the surroundings. [Figure 2] This diagram illustrates the general process by which an information processing system associates an electronic whiteboard with a meeting device. [Figure 3] This diagram shows an example configuration of a record information creation system. [Figure 4] This figure shows an example of the hardware configuration of an information processing system and terminal device. [Figure 5] This figure shows an example of a hardware configuration for a meeting device. [Figure 6] This diagram illustrates the imaging range of the meeting device. [Figure 7] This diagram illustrates the process of extracting images from a panoramic image and a speaker image. [Figure 8] This is a hardware configuration diagram of an example of an interactive whiteboard. [Figure 9] This is an example of a functional block diagram that explains the functions of terminal devices, meeting devices, and information processing systems in a record information creation system by dividing them into blocks. [Figure 10] This figure shows an example of video recording information stored in the information storage unit. [Figure 11] This diagram shows an example of meeting information managed by the Communications Management Department. [Figure 12] This figure shows an example of mapping information stored in the mapping information storage unit, where the conference ID and equipment identification information are associated. [Figure 13] This is an example of a functional block diagram that explains the functions of an electronic whiteboard by dividing them into blocks. [Figure 14] This figure shows an example of device identification information stored in the device information storage unit. [Figure 15] This diagram illustrates the object information stored in the object information storage unit. [Figure 16] This figure shows an example of the initial screen displayed after login by an information recording application running on a terminal device. [Figure 17] This figure shows an example of a recording settings screen displayed by an information recording app. [Figure 18] This figure shows an example of a recording screen displayed by an information recording app while recording. [Figure 19] This figure shows an example of a meeting list screen displayed by an information recording app. [Figure 20] This is an example of a sequence diagram illustrating the process by which a recording information creation system associates a meeting device with an electronic whiteboard. [Figure 21] This is an example of a sequence diagram illustrating the procedure or process by which a user logs into an information processing system by operating an information recording application on a terminal device. [Figure 22] This is an example of a flowchart illustrating the operation of an information processing system. [Figure 23] This figure shows an example of a message displayed by an information recording application on a terminal device. [Figure 24] This diagram shows an example of how a two-dimensional code displayed on an interactive whiteboard can be shown. [Figure 25] This diagram shows an example of how a two-dimensional code displayed on an interactive whiteboard can be shown. [Figure 26] It is a diagram showing an example of a method for displaying a two-dimensional code on an electronic blackboard. [Figure 27] It is a diagram showing an example of a method for displaying a two-dimensional code on an electronic blackboard. [Figure 28] It is a diagram showing an example of a method for displaying a two-dimensional code on an electronic blackboard. [Figure 29] It is a diagram showing an example of a method for displaying a two-dimensional code on an electronic blackboard. [Figure 30] It is a diagram showing an example of a method for displaying a two-dimensional code on an electronic blackboard when the camera of the meeting device is a hemispherical camera. [Figure 31] It is a diagram showing an example of a method for displaying a barcode on an electronic blackboard. [Figure 32] It is an example of a sequence diagram showing the procedure for an information recording application to record a panoramic image, a speaker image, and the screen of the application. [Figure 33] It is an example of a functional block diagram that separately explains the functions of a terminal device, a meeting device, and an information processing system in a recording information creation system (Example 2). [Figure 34] It is an example of a functional block diagram that separately explains the functions of an electronic blackboard in blocks (Example 2). [Figure 35] It is a diagram showing an example of the frequency and bit pattern of sound used for transmitting device identification information. [Figure 36] It is a diagram showing an example of the frequency pattern representing one character of device identification information. [Figure 37] It is an example of a sequence diagram explaining the processing performed by a recording information creation system. [Figure 38] It is a diagram showing an example of the correspondence between frequency and bit data in the case of ultrasonic waves. [Figure 39] It is an example of a sequence diagram explaining the processing when a recording information creation system ends the recording in a meeting. [Figure 40] It is a diagram showing an example of a meeting end notification screen displayed on an electronic blackboard. [Figure 41] This is an example of a functional block diagram (Example 4) that explains the functions of terminal devices, meeting devices, and information processing systems in a recording information creation system by dividing them into blocks. [Figure 42] This figure shows an example of an image of the shape of electronic whiteboard 2 detected from a panoramic image. [Figure 43] This is a sequence diagram illustrating the process when a recording information creation system terminates recording during a meeting (Example 4). [Figure 44] This is an example of a functional block diagram (Example 5) that explains the functions of terminal devices, meeting devices, and information processing systems in a recording information creation system by dividing them into blocks. [Figure 45] This is an example of a sequence diagram illustrating the process when a recording information creation system terminates recording during a meeting (Example 5). [Modes for carrying out the invention]

[0009] The following describes an example of an embodiment for carrying out the present invention: an equipment management system and an equipment management method performed by the equipment management system. [Examples]

[0010] <An example of how to create meeting minutes for remote meetings> First, with reference to Figure 1, we will outline the method for creating meeting minutes using panoramic images and application screens. Figure 1 is a diagram illustrating the outline of creating recording information that saves application screens executed during a remote meeting along with panoramic images of the surroundings. As shown in Figure 1, a user at the local site 102 is conducting a remote meeting with another site 101 using the remote conferencing service system 90.

[0011] The recording information creation system 100 of this embodiment creates recording information (meeting minutes) using a horizontal panoramic image (hereinafter referred to as the panoramic image) captured by a meeting device 60 equipped with a microphone and speaker, and a screen created by an application (hereinafter referred to as the app) executed by the terminal device 10. Regarding audio, the recording information creation system 100 combines the audio received by the remote conferencing app 42 and the audio acquired by the meeting device 60 and includes them in the recording information. The outline is described below.

[0012] (1) The terminal device 10 is running the information recording application 41 and the remote conferencing application 42, which will be described later. Other applications, such as one for displaying documents, may also be running. The information recording application 41 transmits the audio output by the terminal device 10 (including the audio received by the remote conferencing application 42 from other locations) to the meeting device 60 (a second device, an example of such a device). The meeting device 60 mixes (combines) the audio it has acquired with the audio from the remote conferencing application 42.

[0013] (2) The meeting device 60 is equipped with a microphone and, based on the direction from which the sound was acquired, processes the image to extract the speaker from the panoramic image and creates a speaker image. The meeting device 60 transmits both the panoramic image and the speaker image to the terminal device 10.

[0014] (3) The information recording application 41 running on the terminal device 10 can display the panoramic image 203 and the speaker image 204. The information recording application 41 combines the panoramic image 203 and the speaker image 204 with any application screen selected by the user (for example, the screen 103 of the remote conferencing application 42). For example, the panoramic image 203, the speaker image 204, and the application screen 103 are combined so that the panoramic image 203 and the speaker image 204 are on the left and the screen 103 of the remote conferencing application 42 is on the right (hereinafter referred to as the combined image 105). Since the process in (3) is executed repeatedly, the combined image 105 becomes a video (hereinafter referred to as the combined image video). The information recording application 41 also combines the audio synthesized with the combined image video to create a video with audio.

[0015] In this embodiment, an example of combining the panoramic image 203, the speaker image 204, and the application screen 103 is described, but the information recording application 41 may save these separately and arrange them on the screen during playback.

[0016] (4) The information recording application 41 accepts editing work (cutting out unnecessary parts by the user) and completes the combined image video. The combined image video constitutes part of the recorded information.

[0017] (5) The information recording application 41 sends the created combined image video (with audio) to the storage service system 70 for storage.

[0018] (6) The information recording application 41 also extracts only the audio from the combined image video (it may also keep the audio before combining) and sends the extracted audio to the information processing system 50. The information processing system 50 sends the audio to the speech recognition service system 80, which converts the audio into text data. The text data also includes data on how many minutes after the start of recording the speaker spoke.

[0019] In the case of real-time text conversion, the meeting device 60 directly transmits audio to the information processing system 50. The information processing system 50 transmits the text data obtained by speech recognition to the information recording application 41 in real time.

[0020] (7) The information processing system 50 additionally stores text data in the storage service system 70 which stores the combined image video. The text data constitutes part of the recorded information.

[0021] Furthermore, the information processing system 50 can perform billing processing for users based on the services they have used. For example, billing is calculated based on the amount of text data, the file size of combined images and videos, processing time, etc.

[0022] In this way, the combined image video displays panoramic images of the surroundings including the user, as well as images of the speaker, and also shows the screen of the application used during the remote meeting, such as the remote conferencing application 42. When participants or non-participants of the remote meeting view the combined image video as meeting minutes, the atmosphere of the remote meeting is recreated with a sense of presence.

[0023] <Process Overview> Next, with reference to Figure 2, the process of associating the electronic whiteboard 2 with the meeting device 60 will be explained. Figure 2 is a diagram illustrating the general process by which the information processing system 50 associates the electronic whiteboard 2 with the meeting device 60.

[0024] (1) Participants bring their meeting devices 60 to a meeting room where an electronic whiteboard 2 (the first device, an example of other devices) is installed and place them on the table. Therefore, the meeting devices 60 are placed near the electronic whiteboard 2.

[0025] (2) The meeting device 60 captures a two-dimensional code on the electronic whiteboard 2 in which the device identification information is embedded, and transmits the image data to the terminal device 10.

[0026] (3) The terminal device 10 detects device identification information from the image data and transmits it to the information processing system 50.

[0027] (4) When the information processing system 50 receives the device identification information, it issues a conference ID and associates the device identification information of the electronic whiteboard 2 with the conference ID.

[0028] The timing of the information processing system 50 issuing the meeting ID is not limited to this; for example, it may be issued in advance by user operation, or it may be issued automatically at the meeting start time in cooperation with the scheduler.

[0029] (5) The information processing system 50 transmits a message to the electronic whiteboard 2 and the terminal device 10 indicating that registration is complete, including the meeting ID. The electronic whiteboard 2 and the terminal device 10 communicate with the information processing system 50 by specifying the meeting ID, thereby associating the data they each handle with the same meeting. The information processing system 50 records the image data (panoramic image and speaker image) and audio data acquired by the meeting device 60, as well as handwritten data on the electronic whiteboard 2, associating them with the meeting ID.

[0030] Thus, in this embodiment, the recording information creation system 100 allows the meeting device 60 to acquire equipment identification information of equipment used in the conference room, and the terminal device 10 to transmit this information to the information processing system 50, thereby enabling the information processing system 50 to associate the equipment with equipment used in the conference. Since the meeting device 60 is connected to the terminal device 10 that communicates with the information processing system 50, the meeting device 60 is also associated with the conference. There is no need for the user to capture a two-dimensional code displayed by the equipment with a camera or to register the equipment and the meeting device 60 with the information processing system 50 for the conference, minimizing the user's workload and enabling the association (making available) of multiple devices (meeting device 60, electronic whiteboard 2). Examples of "making it available" include various processes such as making the electronic whiteboard a device used in meetings, enabling the electronic whiteboard to send handwritten stroke data and images to the information processing system 50, enabling the creation of meeting minutes using the information on the electronic whiteboard, linking the electronic whiteboard to a meeting, enabling data transmission and reception between the electronic whiteboard and the information processing system 50, and including the electronic whiteboard in a cloud-based electronic whiteboard service (a service that allows terminal devices during remote meetings to write strokes on the same screen). Furthermore, when an electronic whiteboard is made available for use in a meeting, it can be prevented from participating in other meetings taking place at the same time. Also, when an electronic whiteboard is made available for use in a meeting, the meeting minutes created by the information processing system can include the display information and writing information (stroke information) from the electronic whiteboard.

[0031] <About Terminology> An application (or app) is software developed and used for a specific function or purpose, and is not the computer's operating system (OS). Applications include native apps and web apps.

[0032] A running app refers to an app in its state from the time it is launched until it is closed. The app does not need to be active (the app in the foreground); it only needs to be running in the background.

[0033] The surrounding image acquired by the meeting device 60 refers to an image captured with a wider field of view horizontally than the normal field of view. In this embodiment, the surrounding image is described using the term panoramic image. A panoramic image is generally an image with a field of view of approximately 180° to 360° horizontally. The meeting device does not necessarily have to capture a panoramic image with a single unit; it may be a combination of multiple imaging devices with normal fields of view.

[0034] Recorded information refers to the information recorded by the information recording application 41. If the information recording application 41 records the screen of the remote conferencing application 42, the recorded information may become meeting minutes. Recorded information includes, for example, combined video images (including audio) and text data from which the audio has been recognized.

[0035] A tenant is a group of users (companies, local governments, and some of these organizations) that have contracted to receive services from a service provider. The creation of record information and its conversion to text data in this embodiment are performed because the tenant has a contract with the service provider.

[0036] Remote communication refers to communication with someone located at a physically distant location, using software and terminal devices to communicate via voice and video. An example of remote communication is a remote conference, which can also be called a meeting, discussion, gathering, or assembly, while a seminar can be called a training session, study group, workshop, or workshop.

[0037] A base of operations refers to a place that serves as the foundation for activities. An example of a base of operations is a conference room. A conference room is a room set up primarily for the purpose of holding meetings.

[0038] Speech refers to human speech sounds and ambient sounds, while speech data is the data representation of speech. However, in this embodiment, we will not strictly distinguish between the two.

[0039] The first device can be any device that displays information. In this embodiment, it will be described as an electronic blackboard. An electronic blackboard may also be called an electronic whiteboard, electronic information board, etc. A projector is known as a device equivalent to electronic blackboard 2. In addition, the first device may be a digital signage, television, display, multifunction printer, video conferencing terminal, etc.

[0040] Communication-related information is information recorded during communication such as meetings, and includes, for example, information displayed on an electronic whiteboard, images captured by an imaging device, and the voices of speakers. Examples of devices that process communication-related information include electronic whiteboards 2 and meeting devices 60. Meeting devices 60 only need to have an imaging device.

[0041] The same identification information transmitted to the first device and the terminal device is information that associates multiple devices in communication. This same identification information can be any information, but in this embodiment, it is described as a conference ID.

[0042] <Example System Configuration> Next, the system configuration of the recording information creation system 100 will be explained with reference to Figure 3. Figure 3 shows an example configuration of the recording information creation system 100. In Figure 3, one of several locations where remote meetings are held (local location 102) is shown, and the terminal device 10 at local location 102 communicates with the information processing system 50, the storage service system 70, and the remote meeting service system 90 via the network. A meeting device 60 and an electronic whiteboard 2 are also located at local location 102, and the terminal device 10 is connected to the meeting device 60 via a USB cable or the like for communication. The meeting device 60, the electronic whiteboard 2, and the information processing system 50 operate as an equipment management system.

[0043] Terminal device 10 operates at least an information recording application 41 and a remote conferencing application 42. The remote conferencing application 42 can communicate with terminal devices 10 at other locations 101 via a remote conferencing service system 90 on the network, allowing users at each location to hold meetings remotely. The information recording application 41 creates recording information for remote meetings conducted by the remote conferencing application 42 using the functions of the information processing system 50 and the meeting device 60.

[0044] In this embodiment, we describe an example of creating recorded information during a remote meeting, but the meeting does not have to be one that communicates with remote locations. In other words, the meeting may be one in which only participants from a single location are present. In this case, the images captured and the audio collected by the meeting device 60 are saved separately without being combined, and there is no change in the processing of the information recording application 41.

[0045] The terminal device 10 has a built-in camera with a normal field of view (it may also be external), which captures an image of the front, including the user 107 operating the terminal device 10. A normal field of view is an image that is not a panoramic image, but in this embodiment, it is mainly a planar image that is not curved like a 360-degree image. Therefore, the user can conduct a conventional remote conference using the remote conference application 42 without being aware of the information recording application 41. The information recording application 41 and the meeting device 60 do not affect the remote conference application 42, except for an increase in the processing load of the terminal device 10. The remote conference application 42 can also transmit the panoramic image and speaker image captured by the meeting device 60 to the remote conference service system 90.

[0046] The information recording application 41 communicates with the meeting device 60 to create recorded information. The information recording application 41 also performs tasks such as combining audio acquired by the meeting device 60 with audio received by the remote conferencing application 42 from other locations. The meeting device 60 is a meeting device equipped with a panoramic image capture device, a microphone, and a speaker. While the camera on the terminal device 10 can only capture a limited area in front of it, the meeting device 60 can capture the entire surrounding area (not necessarily all around) of the meeting device 60. The meeting device 60 can always keep multiple participants 106, as shown in Figure 3, within its field of view.

[0047] In addition, the meeting device 60 performs tasks such as extracting speaker images from panoramic images. The meeting device 60 is not limited to being placed on a desk; it can be placed anywhere within the site 102. Since the meeting device 60 can capture a full 360-degree image, it may also be placed on the ceiling, for example.

[0048] The information recording application 41 displays a list of applications running on the terminal device 10, performs image synthesis (creation of combined image videos) for the recorded information described above, plays the combined image videos, accepts editing requests, etc. The information recording application 41 also displays a list of remote meetings that have been held or are scheduled to be held. The list of remote meetings is used for information related to the recorded information, allowing the user to link remote meetings with the recorded information.

[0049] The remote conferencing application 42 handles communication connections with other locations 101, sending and receiving images and audio with other locations 101, displaying images, outputting audio, etc.

[0050] The information recording application 41 and the remote conferencing application 42 may be either web applications or native applications. A web application is an application in which a program on a web server and a program on a web browser work together to perform processing, and does not require installation on the terminal device 10. A native application is an application that is installed on the terminal device 10 and used thereafter. In this embodiment, both will be described as native applications.

[0051] The terminal device 10 may be a general-purpose information processing device equipped with communication functions, such as a PC (Personal Computer), smartphone, or tablet device. Other examples of terminal devices 10 include an electronic whiteboard 2, a game console, a PDA (Personal Digital Assistant), a wearable PC, a car navigation system, industrial machinery, medical equipment, or network-connected home appliances. The terminal device 10 only needs to be a device on which the information recording application 41 and the remote conferencing application 42 run.

[0052] The electronic whiteboard 2 displays data handwritten on a touch panel using input methods such as a pen or finger. The electronic whiteboard 2 can communicate with terminal devices 10 etc. via wired or wireless connection and can capture the screen displayed by terminal devices 10 and display it on the display. The electronic whiteboard 2 can convert handwritten data into text data and share information displayed on the display with electronic whiteboards 2 at other locations. The electronic whiteboard 2 may also be a simple whiteboard without a touch panel on which a projector projects images. Alternatively, the electronic whiteboard 2 may be a tablet device, notebook PC (Personal Computer), PDA (Personal Digital Assistant), game console, etc., equipped with a touch panel.

[0053] The electronic whiteboard 2 can communicate with the information processing system 50. For example, after being powered on, the electronic whiteboard 2 can receive information from the information processing system 50 by polling the system.

[0054] The information processing system 50 consists of one or more information processing devices located on a network. The information processing system 50 has one or more server applications that perform processing in cooperation with the information recording application 41, and infrastructure services. These server applications manage lists of remote meetings, manage recorded information from remote meetings, manage various settings and storage paths, etc. The infrastructure services handle user authentication, contracts, billing, etc.

[0055] Furthermore, all or part of the functions of the information processing system 50 may reside in a cloud environment or in an on-premises environment. The information processing system 50 may consist of multiple server devices or a single information processing device. For example, server applications and infrastructure services may be provided by separate information processing devices, and furthermore, there may be separate information processing devices for each function within the server application. The information processing system 50 may be integrated with the storage service system 70 and the speech recognition service system 80 described below.

[0056] The storage service system 70 is a storage means on a network and provides a storage service that accepts the storage of files, etc. Examples of storage service systems 70 include OneDrive®, Google Workspace®, and Dropbox®. The storage service system 70 may also be an on-premises NAS (Network Attached Storage).

[0057] The speech recognition service system 80 provides a service that performs speech recognition on speech data and converts it into text data. The speech recognition service system 80 may be a general-purpose commercial service or it may be part of the functions of the information processing system 50.

[0058] <Example Hardware Configuration> Referring to Figure 4, the hardware configuration of the information processing system 50 and terminal device 10 according to this embodiment will be described.

[0059] <<Information Processing Systems and Terminal Devices>> Figure 4 shows the hardware configuration of an example of the information processing system 50 and terminal device 10 according to this embodiment. As shown in Figure 4, the information processing system 50 and terminal device 10 are built using a computer and include a CPU 501, ROM 502, RAM 503, HD (Hard Disk) 504, HDD (Hard Disk Drive) controller 505, display 506, external device connection I / F (Interface) 508, network I / F 509, bus line 510, keyboard 511, pointing device 512, optical drive 514, and media I / F 516.

[0060] Of these, the CPU 501 controls the operation of the entire information processing system 50 and terminal device 10. The ROM 502 stores programs used to drive the CPU 501, such as IPL. The RAM 503 is used as the work area for the CPU 501. The HD 504 stores various data such as programs. The HDD controller 505 controls the reading or writing of various data to the HD 504 according to the control of the CPU 501. The display 506 displays various information such as cursors, menus, windows, characters, or images. The external device connection I / F 508 is an interface for connecting various external devices. In this case, external devices include, for example, USB (Universal Serial Bus) memory and printers. The network I / F 509 is an interface for data communication using a network. The bus line 510 is an address bus, data bus, etc., for electrically connecting each component such as the CPU 501 shown in Figure 4.

[0061] The keyboard 511 is a type of input means equipped with multiple keys used for inputting characters, numbers, or various instructions. The pointing device 512 is a type of input means used for selecting and executing various instructions, selecting processing targets, moving the cursor, etc. The optical drive 514 controls the reading or writing of various data to the optical storage medium 513, which is an example of a removable recording medium. The optical storage medium 513 may be a CD, DVD, Blu-ray (registered trademark), etc. The media I / F 516 controls the reading or writing (storage) of data to the recording medium 515, such as flash memory.

[0062] <<Meeting Devices>> The hardware configuration of the meeting device 60 will be explained using Figure 5. Figure 5 is an example of a hardware configuration diagram for a meeting device 60 capable of capturing 360° video. In the following, the meeting device 60 is assumed to be a device that uses an image sensor to capture 360° video around the device at a predetermined height, but it may have one or more image sensors. Furthermore, it does not necessarily have to be a dedicated device; a 360° video capture unit can be attached to a PC, digital camera, smartphone, etc., to achieve substantially the same function.

[0063] As shown in Figure 5, the meeting device 60 consists of an imaging unit 601, an image processing unit 604, an imaging control unit 605, a microphone 608, a sound processing unit 609, a CPU (Central Processing Unit) 611, a ROM (Read Only Memory) 612, an SRAM (Static Random Access Memory) 613, a DRAM (Dynamic Random Access Memory) 614, an operation unit 615, an external device connection interface 616, a communication unit 617, an antenna 617a, a sound sensor 618, and a recessed terminal 621 for Micro USB.

[0064] Of these, the imaging unit 601 includes a wide-angle lens (a so-called fisheye lens) 602 with a 360° field of view for forming a hemispherical image, and an image sensor 603 provided in conjunction with each wide-angle lens. The image sensor 603 includes an image sensor such as a CMOS (Complementary Metal Oxide Semiconductor) sensor or a CCD (Charge Coupled Device) sensor that converts the optical image from the fisheye lens 602 into electrical signal image data and outputs it, a timing generation circuit that generates horizontal or vertical synchronization signals and pixel clocks for this image sensor, and a group of registers in which various commands and parameters necessary for the operation of this image sensor are set.

[0065] The image sensor 603 of the imaging unit 601 is connected to the image processing unit 604 via a parallel I / F bus. On the other hand, the image sensor 603 of the imaging unit 601 is connected to the imaging control unit 605 via a serial I / F bus (such as an I2C bus). The image processing unit 604, the imaging control unit 605, and the sound processing unit 609 are connected to the CPU 611 via bus 610. Furthermore, ROM 612, SRAM 613, DRAM 614, operation unit 615, external device connection I / F 616, communication unit 617, and sound sensor 618 are also connected to bus 610.

[0066] The image processing unit 604 receives image data output from the image sensor 603 via a parallel I / F bus, performs predetermined processing on the image data, and creates panoramic images and speaker images from the fisheye video. Furthermore, the image processing unit 604 combines the panoramic image and speaker images, etc., to output a single video.

[0067] The imaging control unit 605 generally uses the I2C bus to set commands and other information in the registers of the image sensor 603, with the imaging control unit 605 acting as the master device and the image sensor 603 as the slave device. It receives necessary commands and other information from the CPU 611. The imaging control unit 605 also uses the I2C bus to acquire status data and other information from the registers of the image sensor 603 and send it to the CPU 611.

[0068] Furthermore, the imaging control unit 605 instructs the image sensor 603 to output image data when the image capture start button on the operation unit 615 is pressed or when an image capture start instruction is received from the PC. Depending on the meeting device 60, there may also be functions to display a preview or video on a display (for example, the display of a PC or smartphone). In this case, the image data output from the image sensor 603 is performed continuously at a predetermined frame rate (frames / minute).

[0069] Furthermore, as will be described later, the imaging control unit 605 also functions as a synchronization control means that works in cooperation with the CPU 611 to synchronize the output timing of image data from the image sensor 603. In this embodiment, the meeting device 60 is not provided with a display, but a display unit may be provided.

[0070] Microphone 608 converts sound into sound (signal) data. The sound processing unit 609 takes in the audio data output from microphones 608a, 608b, and 608c through the I / F bus, mixes this audio data, and performs predetermined processing. The sound processing unit 609 also determines the direction of the sound source (speaker) from the audio level (volume) input from microphones 608a, 608b, and 608c.

[0071] The CPU 611 controls the overall operation of the meeting device 60 and performs necessary processing. The ROM 612 stores various programs for operating the meeting device 60. The SRAM 613 and DRAM 614 are work memories that store programs executed by the CPU 611 and data in progress. In particular, the DRAM 614 stores image data in progress and processed equirectangular projection image data from the image processing unit 604.

[0072] The control unit 615 is a collective term for the control buttons, such as the image capture start button 615a. By operating the control unit 615, the user can start imaging and recording, turn the power on / off, establish a communication connection, and input various settings such as imaging modes and imaging conditions.

[0073] The external device connection interface 616 is an interface for connecting various external devices. In this case, external devices include, for example, a PC (Personal Computer). Video data and image data stored in DRAM 614 are transmitted to an external terminal or recorded on external media via this external device connection interface 616.

[0074] The communication unit 617 may communicate with a cloud server via the internet using wireless communication technology such as Wi-Fi through an antenna 617a provided on the meeting device 60, and transmit the stored video data and image data to the cloud server. The communication unit 617 may also be able to communicate with nearby devices using short-range wireless communication technology such as BLE (Bluetooth Low Energy; registered trademark) or NFC.

[0075] The audio sensor 618 is a sensor that acquires 360° audio information in order to determine from which direction loud audio is being input within a 360° radius (horizontal plane) around the meeting device 60. The audio processing unit 609 identifies the direction of the strongest sound based on the input 360° audio parameters and outputs the audio input direction within the 360°.

[0076] Furthermore, other sensors (such as compass / accelerometers or GPS) can be used to calculate direction, position, angle, acceleration, etc., and these can be used for image correction or to add location information.

[0077] The image processing unit 604 also creates a panoramic image in the following way: The CPU 611 performs predetermined camera image processing, such as Bayer conversion (RGB interpolation), on the RAW data input from the image sensor that inputs a spherical image to create a fisheye image (curved image). Furthermore, it performs DeWarp processing (distortion correction processing) on ​​the created fisheye image (curved image) to create a panoramic image (flat image) that captures 360° around the meeting device 60.

[0078] The CPU 611 creates speaker images in the following way: The CPU 611 creates speaker images by extracting the speaker from a panoramic image (flat image) that captures the surrounding 360°. The CPU 611 uses the direction of the audio input, identified from the 360° output by the audio sensor 618 and the audio processing unit 609, as the direction of the speaker, and extracts the speaker image from the panoramic image. In this case, the method for extracting a person's image from the direction of the audio input involves cutting out a 30° section centered on the audio direction identified from the 360° section, and then performing face detection within that section to extract the image. The CPU 611 further identifies speaker images for a specific number of people (e.g., 3 people) who have spoken most recently from among the extracted speaker images.

[0079] The panoramic image and the images of one or more speakers may be sent individually to the information recording application 41, or the meeting device 60 may create a single image from them and send it to the information recording application 41. In this embodiment, the panoramic image and the images of one or more speakers are sent individually from the meeting device 60 to the information recording application 41.

[0080] Figure 6 illustrates the imaging range of the meeting device 60. As shown in Figure 6(a), the meeting device 60 images a 360° range in the horizontal direction. As shown in Figure 6(b), the meeting device 60 has an imaging range that is 0° horizontal to the height of the meeting device 60, and a predetermined angle above and below it.

[0081] Figure 7 illustrates the extraction of panoramic and speaker images. As shown in Figure 7, the image captured by the meeting device 60 forms a part 110 of a sphere and therefore has a three-dimensional shape. As shown in Figure 6(b), the meeting device 60 divides the field of view into predetermined angles vertically and horizontally and performs perspective projection transformation. By performing perspective projection transformation seamlessly across the entire 360° horizontal direction, a predetermined number of planar images are obtained, and by stitching these predetermined number of planar images together horizontally, a panoramic image 111 is obtained. The meeting device 60 also performs face detection from the panoramic image within a predetermined range centered on the direction of sound, and creates speaker images 112 by extracting 15° to the left and right (30° in total) from the center of the face.

[0082] <<Electronic Whiteboard>> Figure 8 is a hardware configuration diagram of the electronic whiteboard 2. As shown in Figure 8, the electronic whiteboard 2 is equipped with a CPU (Central Processing Unit) 401, ROM (Read Only Memory) 402, RAM (Random Access Memory) 403, SSD (Solid State Drive) 404, network I / F 405, and external device connection I / F (Interface) 406.

[0083] Of these components, the CPU 401 controls the overall operation of the electronic whiteboard 2. The ROM 402 stores programs used to start the OS, such as the IPL (Initial Program Loader). The RAM 403 is used as the work area for the CPU 401. The SSD 404 stores various data, such as programs for the electronic whiteboard 2. The network interface 405 controls communication with the communication network. The external device connection interface 406 is an interface for connecting various external devices. In this case, external devices include, for example, a USB (Universal Serial Bus) memory 430 and external devices (microphone 440, speaker 450, camera 460).

[0084] Furthermore, the electronic whiteboard 2 is equipped with a capture device 411, a GPU 412, a display controller 413, a contact sensor 414, a sensor controller 415, an electronic pen controller 416, a short-range communication circuit 419, and an antenna 419a for the short-range communication circuit 419, a power switch 422, and selection switches 423.

[0085] Of these, the capture device 411 displays the information shown on the external PC (Personal Computer) 470's display as a still image or video. The GPU (Graphics Processing Unit) 412 is a semiconductor chip specializing in graphics. The display controller 413 controls and manages the screen display in order to output the output image from the GPU 412 to the display 480, etc. The contact sensor 414 detects when an electronic pen 490 or the user's hand H, etc., touches the display 480. The sensor controller 415 controls the processing of the contact sensor 414. The contact sensor 414 performs coordinate input and coordinate detection using an infrared blocking method. This method of coordinate input and detection involves two light-emitting and receiving devices installed at both ends of the upper side of the display 480 emitting multiple infrared rays parallel to the display 480, and receiving the light that is reflected by reflective members provided around the display 480 and returns along the same optical path as the light emitted by the light-receiving element. The light-receiving and receiving device, which is a contact sensor 414, outputs the position where infrared light is blocked by an object (position on the light-receiving element) to the sensor controller 415, and the sensor controller 415 identifies the coordinate position of the object's contact point from these two positional pieces of information. The electronic pen controller 416 communicates with the electronic pen 490 via Bluetooth to determine whether the pen tip or the pen end has touched the display 480. The near-field communication circuit 419 is a communication circuit such as NFC (Near Field Communication) or Bluetooth (registered trademark). The power switch 422 is a switch for turning the power of the electronic whiteboard 2 ON / OFF. The selection switches 423 are a group of switches for adjusting, for example, the brightness and color of the display 480.

[0086] Furthermore, the electronic whiteboard 2 is equipped with a bus line 410. The bus line 410 is an address bus, data bus, etc., for electrically connecting each component, such as the CPU 401 shown in Figure 8.

[0087] Furthermore, the contact sensor 414 is not limited to an infrared blocking type; it may also be a capacitive type touch panel that identifies the contact position by detecting a change in capacitance. The contact sensor 414 may also be a resistive type touch panel that identifies the contact position by the voltage change between two opposing resistive films. The contact sensor 414 may also be an electromagnetic induction type touch panel that identifies the contact position by detecting electromagnetic induction caused by the contact of an object to the display. In addition, the contact sensor 414 may use various detection means. Moreover, the electronic pen controller 416 may determine whether there is a touch not only on the tip and end of the electronic pen 490, but also on the part of the electronic pen 490 that the user holds and other parts of the electronic pen.

[0088] <About the features> Next, with reference to Figure 9, the functional configuration of the recording information creation system 100 will be described. Figure 9 is an example of a functional block diagram that explains the functions of the terminal device 10, meeting device 60, and information processing system 50 in the recording information creation system 100 by dividing them into blocks.

[0089] <<Terminal device>> The information recording application 41 operating on the terminal device 10 includes a communication unit 11, an operation reception unit 12, a display control unit 13, an application screen acquisition unit 14, an audio acquisition unit 15, a device communication unit 16, a recording control unit 17, an audio data processing unit 18, a recording playback unit 19, an upload unit 20, an editing processing unit 21, a code analysis unit 22, and a time measurement unit 25. Each of these units in the terminal device 10 is a function or means of function realized by any of the components shown in Figure 4 operating according to instructions from the CPU 501 in accordance with the information recording application 41 deployed from the HD 504 to the RAM 503. The terminal device 10 also has a storage unit 1000 constructed from the HD 504, etc., as shown in Figure 4. An information storage unit 1001 is constructed in the storage unit 1000.

[0090] The communication unit 11 communicates various types of information with the information processing system 50 via the network. For example, the communication unit 11 receives a list of remote meetings from the information processing system 50 and sends voice data recognition requests to the information processing system 50.

[0091] The display control unit 13 displays various screens that serve as the user interface in the information recording application 41 according to the screen transitions set in the information recording application 41. The operation reception unit 12 receives various operations on the information recording application 41.

[0092] The application screen acquisition unit 14 acquires the desktop screen or the screen of the application selected by the user from the OS (Operating System), etc. If the application selected by the user is the remote conferencing application 42, the screen generated by the remote conferencing application 42 (images of each location, images of documents, etc.) is obtained.

[0093] The audio acquisition unit 15 acquires audio data received by the terminal device 10 from the remote conferencing application 42 during a remote conference. Note that the audio acquired by the audio acquisition unit 15 does not include the audio collected by the terminal device 10, but only the audio data received during the remote conference. This is because the meeting device 60 collects audio separately.

[0094] The device communication unit 16 communicates with the meeting device 60 using a USB cable or the like. The device communication unit 16 may communicate with the meeting device 60 using wireless LAN, Bluetooth (registered trademark), etc. The device communication unit 16 receives panoramic images and speaker images from the meeting device 60 and transmits the audio data acquired by the audio acquisition unit 15 to the meeting device 60. The device communication unit 16 receives the audio data synthesized by the meeting device 60.

[0095] The recording control unit 17 combines the panoramic image and speaker image received by the device communication unit 16, and the application screen acquired by the application screen acquisition unit 14, to create a combined image. The recording control unit 17 also repeatedly connects the created combined images in chronological order to create a combined image video, and combines the audio data synthesized by the meeting device 60 into the combined image video to create a combined image video with audio.

[0096] The audio data processing unit 18 requests the information processing system 50 to convert the audio data combined with the combined image video extracted by the video recording control unit 17, or the synthesized audio data received from the meeting device 60, into text data.

[0097] The recording and playback unit 19 plays back the combined image video. The combined image video is stored in the terminal device 10 during recording and then uploaded to the information processing system 50.

[0098] When the remote conference ends, the upload unit 20 transmits the combined image and video to the information processing system 50.

[0099] The editing processing unit 21 performs editing of the combined image video (such as deleting parts or splicing them together) in response to user operations.

[0100] The code analysis unit 22 detects two-dimensional codes contained in the panoramic image and analyzes the two-dimensional codes to obtain device identification information.

[0101] The time measurement unit 25 measures the time from when the information recording application 41 is launched until a two-dimensional code is received from the meeting device 60. After a certain period of time has elapsed, the time measurement unit 25 notifies the display control unit 13, and the display control unit 13 displays an error dialog.

[0102] Figure 10 shows the video recording information stored in the information storage unit 1001. The video recording information includes items such as meeting ID, recording ID, update date and time, title, upload, and storage location. When a user logs in to the information processing system 50, the information recording application 41 downloads meeting information from the meeting information storage unit 5001 of the information processing system 50. The meeting ID and other information included in the meeting information are reflected in the video recording information. The video recording information in Figure 10 is held by a terminal device 10 operated by a particular user.

[0103] The meeting ID is identification information that identifies a remote meeting that has been held. The meeting ID is assigned when the remote meeting is scheduled in the meeting management system 9, or it is assigned by the information processing system 50 upon request from the information recording application 41.

[0104] The recording ID is identification information that identifies the combined video image recorded during a remote meeting. The recording ID is assigned by the meeting device 60, but it may also be assigned by the information recording application 41 or the information processing system 50. The same meeting ID may be assigned to a different recording ID if the recording of the remote meeting was stopped midway through but resumed for some reason.

[0105] • The update date and time is the date and time when the combined image / video was updated (recording ended). If the combined image / video was edited, it will be the date and time of editing.

[0106] The title is the name of the meeting. This may be set when registering the meeting in the meeting management system 9, or it may be set arbitrarily by the user.

[0107] • Upload indicates whether or not the combined image video has been uploaded to the information processing system 50.

[0108] The storage location indicates the location (URL or file path) where the combined image / video and text data are stored in the storage service system 70. Therefore, users can view the uploaded combined image / video at their discretion. Note that the combined image / video and text data are saved with separate filenames, for example, following the URL.

[0109] <<Meeting Devices>> Let's return to Figure 9 for explanation. The meeting device 60 includes a terminal communication unit 61, a panoramic image creation unit 62, a speaker image creation unit 63, a sound collection unit 64, and a speech synthesis unit 65. Each of these units in the meeting device 60 is a function or means of functioning, realized by any of the components shown in Figure 5 operating according to instructions from the CPU 611 in accordance with a program expanded from the ROM 612 to the DRAM 614.

[0110] The terminal communication unit 61 communicates with the terminal device 10 using a USB cable or the like. The terminal communication unit 61 does not only need to be connected by a wired cable, but it is also acceptable if it can communicate with the terminal device 10 via wireless LAN, Bluetooth (registered trademark), etc.

[0111] The panoramic image creation unit 62 creates a panoramic image. The speaker image creation unit 63 creates a speaker image. The methods for creating these images are explained in Figures 6 and 7. The panoramic image creation unit 62 also serves as the image data acquisition unit.

[0112] The sound collection unit 64 converts the audio acquired by the microphone of the meeting device 60 into audio data (digital). This allows the content spoken by users and participants at the terminal device 10 to be collected.

[0113] The speech synthesis unit 65 synthesizes the speech transmitted from the terminal device 10 with the speech collected by the sound collection unit 64. This combines speech spoken at other locations 101 with speech spoken at the local location 102.

[0114] <<Information Processing System>> The information processing system 50 includes a communication unit 51, an authentication unit 52, a screen generation unit 53, a communication management unit 54, an equipment management unit 55, and a text conversion unit 56. Each of these units in the information processing system 50 is a function or means of functioning, realized by any of the components shown in Figure 4 operating according to instructions from the CPU 501 in accordance with a program deployed from the HD 504 to the RAM 503. The information processing system 50 also has a storage unit 5000 constructed from the HD 504 and other components shown in Figure 4. The storage unit 5000 includes a conference information storage unit 5001, a recording information storage unit 5002, and a mapping information storage unit 5003.

[0115] The communication unit 51 sends and receives various types of information with the terminal device 10. For example, the communication unit 51 may send a list of remote meetings to the terminal device 10 or receive a voice data recognition request from the terminal device 10.

[0116] The authentication unit 52 authenticates the user operating the terminal device 10. The authentication unit 52 authenticates the user, for example, by checking whether the authentication information (user ID and password) included in the authentication request received by the communication unit 51 matches the authentication information it has previously stored. The authentication information may be an IC card number, biometric authentication information such as a face or fingerprint, etc. The authentication unit 52 may also authenticate using an external authentication system or an authentication method such as OAuth.

[0117] The screen generation unit 53 generates screen information that the terminal device 10 will display in the web application. The screen information is created using HTML, XML, CSS (Cascade Style Sheet), and JavaScript (registered trademark), etc.

[0118] The communication management unit 54 retrieves information about remote meetings from the meeting management system 9 using each user's account or a system account assigned to the information processing system 50. The communication management unit 54 stores the meeting information of scheduled meetings in the meeting information storage unit, associating it with the meeting ID. The communication management unit 54 also retrieves meeting information that users belonging to the tenant have viewing rights to. Since each meeting is assigned a meeting ID, the remote meeting and its recorded information are associated using the meeting ID.

[0119] When the equipment management unit 55 receives equipment identification information for the electronic whiteboard 2 and meeting device 60 used in a meeting, it associates them and stores them in the association information storage unit 5003. Therefore, the meeting ID is associated with the equipment identification information of the electronic whiteboard 2 and the equipment identification information of the meeting device 60. Combined video is also associated with the meeting ID, so handwritten data entered on the electronic whiteboard 2 is also associated with the combined video. Furthermore, when recording ends (when the meeting ends), the equipment management unit 55 deletes the association from the association information storage unit 5003.

[0120] The text conversion unit 56 converts the voice data requested by the terminal device 10 to be converted to text data into text data using an external voice recognition service system 80. The text conversion unit 56 may also perform the conversion itself.

[0121] Figure 11 shows an example of meeting information stored in the meeting information storage unit 5001, which is managed by the communication management unit 54. The communication management unit 54 can use the above account to obtain a list of remote meetings that the user belonging to the tenant has permission to view. In this embodiment, remote meetings are used as an example, but the list of remote meetings also includes meetings held in only one conference room.

[0122] Meeting information is managed using a meeting ID, which is associated with participants, title (meeting name), start date and time, end date and time, location, etc. These are just examples of meeting information; meeting information may include other details as well.

[0123] • The "Participants" section lists the attendees of the meeting.

[0124] The title field should represent the meeting's content, such as the meeting name or agenda.

[0125] The "Start Date and Time" field indicates the scheduled date and time when the meeting will begin.

[0126] The "End Date and Time" field indicates the scheduled end date and time of the meeting.

[0127] The "Location" field specifies the meeting venue, such as the conference room, branch office name, or building.

[0128] The "Electronic Whiteboard Information" item is the identification information for Electronic Whiteboard 2, which was used in the meeting.

[0129] The "Meeting Device" item contains the identification information for the 60 meeting devices used in the meeting.

[0130] As shown in Figures 10 and 11, the combined video footage recorded during the meeting is identified by the meeting ID.

[0131] The recording information stored in the recording information storage unit 5002 may be the same as in Figure 10. However, the information processing system 50 has a list of combined video recordings made by all users belonging to the tenant.

[0132] Figure 12 shows the mapping information stored in the mapping information storage unit 5003, where the device identification information of the electronic whiteboard 2 and the device identification information of the meeting device 60 are associated with the meeting ID. This mapping information is retained from the time the information recording application 41 transmits the device identification information to the information processing system 50 until the recording is terminated.

[0133] <<Electronic Whiteboard>> Figure 13 is a functional block diagram illustrating the functions of the electronic whiteboard 2 in blocks. The electronic whiteboard 2 includes a contact position detection unit 31, a drawing data generation unit 32, a data recording unit 33, a display control unit 34, a code generation unit 35, and a communication unit 36. Each function of the electronic whiteboard 2 is realized by an instruction from the CPU 401, following a program deployed from the SSD 404 onto the RAM 403, in which any of the components shown in Figure 8 operates.

[0134] The contact position detection unit 31 detects the coordinates of the position where the electronic pen 490 makes contact with the contact sensor 414. The drawing data generation unit 32 obtains the coordinates of the contact point of the pen tip of the electronic pen 490 from the contact position detection unit 31. The drawing data generation unit 32 connects these coordinate points by interpolation to generate stroke data.

[0135] The display control unit 34 displays handwritten data, user-operated menus, and other information on the display.

[0136] The data recording unit 33 stores handwritten data, shapes such as circles and triangles, stamps such as "completed," PC screens, files, etc., written on the electronic whiteboard 2 in the object information storage unit 3002. Handwritten data, shapes, images such as PC screens, files, etc., are treated as objects. For handwritten data, a set of stroke data becomes a single object according to temporal divisions due to interruptions in handwriting and spatial divisions due to differences in handwriting location.

[0137] The communication unit 36 ​​connects to Wi-Fi or LAN and communicates with the information processing system 50. The communication unit 36 ​​transmits object information to the information processing system 50 and receives object information stored in the information processing system 50 and displays it on the display 480.

[0138] The code generation unit 35 generates a two-dimensional code by encoding the device identification information of the electronic whiteboard 2 stored in the device information storage unit 3001 and information indicating that it is a device usable in a meeting into a two-dimensional pattern. The code generation unit 35 can also encode the device identification information of the electronic whiteboard 2 and information indicating that it is a device usable in a meeting into a barcode. The device identification information can be a serial number or a UUID (Universally Unique Identifier), or it can be something set by the user. The code generation unit 35 also functions as an output unit that outputs a two-dimensional code or a barcode.

[0139] Furthermore, the electronic whiteboard 2 has a storage unit 3000 constructed by the SSD 404 shown in Figure 8. The storage unit 3000 has a device information storage unit 3001 and an object information storage unit 3002.

[0140] Figure 14 shows the device identification information and other data stored in the device information storage unit 3001. • The device identification information is the identification information for electronic whiteboard 2. The IP address is the IP address that other devices use to connect to the electronic whiteboard 2 via the network. • The password is used for authentication when other devices connect to the electronic whiteboard 2.

[0141] Figure 15 is a diagram illustrating the object information stored in the object information storage unit 3002. The object information manages the objects displayed by the electronic whiteboard 2. The object information is transmitted to the information processing system 50 and used as meeting minutes. When an electronic whiteboard 2 is located at another site during a remote meeting, the object information is shared.

[0142] The meeting ID is the identification information for the meeting, notified by the information processing system 50.

[0143] • An object ID is an identifier that identifies an object.

[0144] • The type refers to the type of object, which includes handwritten data, text, shapes, images, etc. Handwritten data is stroke data (a sequence of coordinate points). Text is a string of characters (character codes) entered from a software keyboard. Strings of characters are sometimes also called text data. Shapes are geometric shapes such as triangles and squares. Images are image data such as JPEG, PNG, and TIFF files imported from a PC or the internet.

[0145] • One screen of the electronic whiteboard 2 is referred to as a page. The item on the page is its page number.

[0146] The coordinates indicate the position of an object relative to a predetermined origin on the electronic whiteboard 2. The object's position is, for example, the top-left vertex of its bounding rectangle. The coordinates are expressed, for example, in pixels on the display.

[0147] • The size is the width and height of the bounding rectangle of the object.

[0148] <Screen transition> Next, with reference to Figures 16 to 19, several screens displayed by the terminal device 10 during a remote conference will be described. Figure 16 is the initial screen 200 displayed after login by the information recording application 41 running on the terminal device 10. The user of the terminal device 10 connects the information recording application 41 to the information processing system 50. When the user enters authentication information and successfully logs in, the initial screen 200 of Figure 16 is displayed.

[0149] The initial screen 200 has a fixed display button 201, a front view change button 202, a panoramic image 203, one or more speaker images 204a to 204c (hereinafter referred to as speaker image 204 when not distinguished), and a recording start button 205. If the meeting device 60 is already running and capturing images of the surroundings when the user logs in, the panoramic image 203 and speaker images 204 created by the meeting device 60 are displayed on the initial screen 200. Therefore, the user can decide whether or not to start recording while viewing these images. If the meeting device 60 is not running (not capturing images), the panoramic image 203 and speaker images 204 are not displayed.

[0150] The information recording application 41 may display speaker images 204 of all participants based on all faces detected from the panoramic image 203, or it may display only the speaker images 204 of the N people who have spoken most recently. Figure 16 shows an example where up to three speaker images 204 are displayed. There may be no speaker images 204 until a participant speaks (one is added each time a participant speaks), or three speaker images 204 of participants facing a predetermined direction may be displayed (they are swapped depending on the participant's speech).

[0151] If no one is speaking, such as immediately after the meeting device 60 is activated, a speaker image 204 will be created in a predetermined direction within the horizontal 360° (e.g., 0°, 120°, 240°). If a fixed display setting (described later) is enabled, that setting will take precedence.

[0152] The fixed display button 201 is a button that allows the user to perform an operation to fix a certain area of ​​the panoramic image 203 as the speaker image 204 and zoom in on it.

[0153] The front view change button 202 is a button for the user to change the front view of the panoramic image 203 (since the panoramic image captures 360° horizontally, the right and left edges coincide). The user can slide the panoramic image 203 left or right with a pointing device to determine which participant is in the front view. The user's operation is sent to the meeting device 60, which changes the angle that is considered the front view out of the 360° horizontal direction to create a panoramic image and sends it to the terminal device 10.

[0154] When the user presses the recording start button 205, the information recording application 41 displays the recording settings screen 210 shown in Figure 17.

[0155] Figure 17 shows an example of the recording settings screen 210 displayed by the information recording application 41. On the recording settings screen 210, the user can set whether to record (include in recording) the panoramic image and speaker image created by the meeting device 60, as well as the desktop screen or running application screen of the terminal device 10. If the information recording application 41 does not record either the panoramic image and speaker image, or the desktop screen or running application screen, only the audio (audio output by the terminal device 10 + audio collected by the meeting device 60) will be recorded.

[0156] The camera toggle button 211 is a button that switches the recording of panoramic images and speaker images created by the meeting device 60 on and off. The camera toggle button 211 may also be configured to record panoramic images and speaker images individually.

[0157] The PC screen toggle button 212 is a button that switches the recording of the desktop screen of the terminal device 10 and the screen of an application running on the terminal device 10 on and off. When the PC screen toggle button 212 is on, the desktop screen is recorded.

[0158] If a user wants to record an app screen, they further select an app from the app selection field 213. The app selection field 213 displays the names of apps currently running on the terminal device 10 in a dropdown format. The user can select the app to record. This app name is obtained by the information recording app 41 from the OS. The information recording app 41 can display only apps that have a UI (screen) among the running apps. The remote conferencing app 42 may be included in the selected apps. Therefore, the information recording app 41 can also record the materials displayed by the remote conferencing app 42 and the participants at each location in video. In addition, the apps displayed in the dropdown are diverse, including presentation apps, word processing apps, spreadsheet apps, and web browser apps. Therefore, the user can flexibly select the app screens to include in the combined video.

[0159] Furthermore, when recording on an app-by-app basis, users can select multiple apps. The information recording app 41 can record the screens of all selected apps.

[0160] If both the camera toggle button 211 and the PC screen toggle button 212 are turned off, the recording content confirmation window 214 will display "Only audio will be recorded." This audio consists of the audio output by the terminal device 10 (audio received by the remote conferencing application 42 from other locations 101) and the audio collected by the meeting device 60. In other words, if a remote conference is being held, the audio from the remote conferencing application 42 and the audio from the meeting device 60 will be saved regardless of image recording. However, the user may selectively stop saving the audio from the remote conferencing application 42 and the audio from the meeting device 60 in their settings.

[0161] Depending on the on / off combination of the camera toggle button 211 and the PC screen toggle button 212, the combined image video is recorded as follows. The combined image video is also displayed in real time in the recording content confirmation window 214.

[0162] If the camera toggle button 211 is on and the PC screen toggle button 212 is off, the panoramic image captured by the meeting device 60 and the speaker image will be displayed in the recording content confirmation window 214.

[0163] If the camera toggle button 211 is off and the PC screen toggle button 212 is on (screen is also selected), the desktop screen or the screen of the selected application will be displayed in the recording content confirmation window 214.

[0164] When the camera toggle button 211 is on and the PC screen toggle button 212 is on, the recording content confirmation window 214 displays the panoramic image and speaker image captured by the meeting device 60, as well as the desktop screen and the screen of the selected application, side by side.

[0165] Therefore, there are cases where the panoramic image, speaker image, and app screen are not recorded, or where none of the panoramic image, speaker image, and app screen are recorded at all. However, in this embodiment, for convenience, the images created by the information recording app 41 are referred to as a combined image video.

[0166] Furthermore, the recording settings screen 210 has a checkbox 215 along with the message, "Automatically transcribe the recording after uploading it." The recording settings screen 210 also has a "Start Recording Now" button 216. When the user checks the checkbox 215, the recorded video is attached with text data containing the spoken words from the remote meeting. In this case, after the recording is finished, the information recording application 41 uploads the audio to the information processing system 50 along with a request for conversion to text data. Also, when the user presses the "Start Recording Now" button 216, the recording screen 220 shown in Figure 18 is displayed.

[0167] Figure 18 shows an example of the recording screen 220 displayed by the information recording application 41 during recording. Note that the explanation of Figure 18 mainly focuses on the differences from Figure 16. The recording screen 220 displays the combined image video being recorded in real time, based on the conditions set by the user in the recording settings screen 210. In Figure 18, the recording screen 220 is shown with the camera toggle button 211 on and the PC screen toggle button 212 off, displaying the panoramic image 203 and speaker image 204 (both videos) created by the meeting device 60. The recording screen 220 also displays the recording icon 225, the pause button 226, and the recording stop button 227.

[0168] The pause button 226 is used to stop recording, and after stopping, it also accepts requests to resume recording. The recording end button 227 is used to end recording. The recording ID does not change when the pause button 226 is pressed, but it does change when the recording end button 227 is pressed. After pausing, the user can also set the recording conditions that were set on the recording settings screen 210 again when resuming recording. In that case, the information recording application 41 may create multiple recording files each time recording is stopped (for example, when the recording end button 227 is pressed), or it may combine multiple files to form a single continuous video (for example, when the pause button 226 is pressed). Also, when the information recording application 41 plays a combined video, it may play multiple recording files continuously as a single video.

[0169] The recording screen 220 also has a "Get Information from Calendar" button 221, a meeting name field 222, a time field 223, and a location field 224. The "Get Information from Calendar" button 221 is a button for the user to retrieve meeting information from the meeting management system 9. When the "Get Information from Calendar" button 221 is pressed, the information recording application 41 retrieves and displays a list of meetings that the user has permission to view from the information processing system 50. The user selects the remote meeting to be held from the meeting list. As a result, the meeting information is reflected in the meeting name field 222, the time field 223, and the location field 224. The meeting name field 222 displays the title of the meeting information, the time field 223 displays the start and end times, and the location field 224 displays the location. In addition, the meeting information and recording information in the meeting management system are associated with each other by the meeting ID.

[0170] Once the remote meeting ends and the user stops recording, a combined video with audio is created.

[0171] Figure 19 shows an example of the meeting list screen 230 displayed by the information recording application 41. The meeting list screen 230 is a list of meetings, and can display a list of recorded information from remote meetings. It also includes meetings held only within a specific meeting room, regardless of whether they were remote meetings. The meeting list screen 230 displays meeting information that the logged-in user has permission to view in the meeting information storage unit 5001. Information from video recordings stored in the information storage unit 1001 may be further integrated.

[0172] The meeting list screen 230 is displayed when the user selects the meeting list tab 231 on the initial screen 200 in Figure 16. The meeting list screen 230 displays a list 236 of the recorded information that this user has permission to view. The meeting creator (minutes creator) can set viewing permissions for participants. The meeting list may be a list of saved recorded information, or a list of meeting schedules or meeting data.

[0173] The meeting list screen 230 has the following items: checkbox 232, update date and time 233, title 234, and status 235.

[0174] Checkbox 232 accepts the selection of recording files. Checkbox 232 is used when the user wants to delete multiple recording files at once.

[0175] • The update time 233 indicates the start time of recording the combined image video. If it has been edited, the edit time is acceptable.

[0176] Title 234 is the title of the meeting (agenda, etc.). It can be copied from the meeting information or set by the user.

[0177] Status 235 indicates whether the combined image / video has been uploaded to the information processing system 50. If it has not been uploaded, "Local PC" is displayed; if it has been uploaded, "Uploaded" is displayed. If it has not been uploaded, the upload button is displayed. If there are unuploaded combined image / videos, the information recording application 41 should automatically upload them when the user logs into the information processing system 50.

[0178] When the user selects a title or other item from the list of combined images and videos 236 using the pointing device, the information recording application 41 displays a recording playback screen, but this is omitted in this embodiment. On the recording playback screen, it is possible to play back the combined images and videos.

[0179] Ideally, users should be able to filter meetings by update date, title, keywords, etc. Furthermore, if the number of displayed meetings is large and it is difficult to find the relevant one, a search function should be available allowing users to input terms to narrow down the recorded information based on keywords found in meeting statements or titles. This search function would enable users to quickly find the desired recorded information, even when dealing with a large volume of records. Additionally, the meeting list screen should allow users to sort meetings by update date or title.

[0180] <Action or process> Figure 20 is a sequence diagram illustrating the process by which the recording information creation system 100 associates the meeting device 60 with the electronic whiteboard 2. In the explanation of Figure 20, a user participating in a meeting is attempting to participate in the meeting using both the meeting device 60 and the electronic whiteboard 2 in the same meeting.

[0181] S1: When the electronic whiteboard 2, which is placed in the conference room used for the meeting, is powered on, it communicates with the pre-configured information processing system 50 and registers that it can be associated with the meeting by specifying the device identification information.

[0182] S2: In addition, the code generation unit 35 of the electronic whiteboard 2, which is placed in the conference room used for meetings, generates a two-dimensional code that encodes the device identification information of the device and that indicates that it is device identification information, and the display control unit 34 displays it. The two-dimensional code may also include a password for the electronic whiteboard 2 to authenticate other devices.

[0183] S3: The user enters the conference room where the electronic whiteboard 2 is installed, bringing the terminal device 10 and the meeting device 60, and connects the terminal device 10 and the meeting device 60 with a USB cable. The meeting device 60 starts up due to power supply from the USB cable or when the power is turned on. This puts the meeting device 60 into standby mode. The user also starts the information recording application 41 on the terminal device 10. When the information recording application 41 starts communicating with the meeting device 60, the meeting device 60 starts capturing images and collecting sound. The panoramic image creation unit 62 of the meeting device 60 creates a panoramic image (image data) that captures the surroundings, including a two-dimensional code.

[0184] S4: The terminal communication unit 61 of the meeting device 60 transmits the panoramic image to the terminal device 10.

[0185] S5: The device communication unit 16 of the terminal device 10 receives a panoramic image, and the code analysis unit 22 detects the two-dimensional code displayed by the electronic whiteboard 2 within the panoramic image. The code analysis unit 22 decodes the two-dimensional code, and if it determines that it contains information indicating that the device is usable for meetings, it obtains the device identification information of the electronic whiteboard 2 from the two-dimensional code. Note that the analysis of the two-dimensional code may also be performed by the meeting device 60.

[0186] S6: The communication unit 11 of the information recording application 41 sends a registration request for the meeting to the information processing system 50, specifying the device identification information of the electronic whiteboard 2. Preferably, the communication unit 11 further sends the identification information of the meeting device 60 to the information processing system 50.

[0187] S7: When the communication unit 51 of the information processing system 50 receives a registration request for a meeting (device identification information), the communication management unit 54 issues a meeting ID. If the information recording application 41 accepts the selection of a meeting from the meeting list screen 230, etc., the communication unit 51 attaches the meeting ID to the device identification information, so the communication management unit 54 does not issue a meeting ID.

[0188] S8: The equipment management unit 55 then associates the equipment identification information with the conference ID, preferably the equipment identification information of the meeting device 60 with the conference ID, and stores it in the association information storage unit 5003.

[0189] S9, S10: The communication unit 51 of the information processing system 50 notifies the terminal device 10 and the electronic whiteboard 2 of the conference ID. The communication unit 11 of the terminal device 10 receives and stores the conference ID. Similarly, the communication unit 36 ​​of the electronic whiteboard 2 receives and stores the conference ID. The terminal device 10 receives at least one of the conference ID or device identification information as a response to the conference registration request. The electronic whiteboard 2 and the information processing system 50 may communicate using a bidirectional communication method such as WebSocket, which enables push-type communication from the information processing system 50 to the electronic whiteboard 2.

[0190] Since the electronic whiteboard 2 and the terminal device 10 have the same conference ID, the electronic whiteboard 2 and the meeting device 60 are associated with the conference. Subsequently, the terminal device 10 attaches at least one of the conference ID or the identification information of the meeting device 60 to the data it transmits, and the electronic whiteboard 2 attaches at least one of the conference ID or the equipment identification information to the data it transmits. In this embodiment, the conference ID is attached to the communication, but even if the equipment identification information or the identification information of the meeting device 60 is attached to the communication, the information processing system 50 can identify the conference ID using the association information.

[0191] In Figure 20, the mapping between the electronic whiteboard 2 and the meeting ID is for the purpose of mapping recorded information to handwritten data, etc. However, electronic whiteboard 2 also has a function to share objects with other electronic whiteboards 2 located in remote locations. Electronic whiteboard 2 and other electronic whiteboards 2 are registered with the remote meeting server, and handwritten data, etc. can be shared in real time.

[0192] Figure 21 is a flowchart illustrating the operation of the information processing system 50. The communication unit 51 of the information processing system 50 waits until it receives device identification information from the terminal device 10 (S101).

[0193] If the communications unit 51 receives device identification information, the communications management unit 54 issues a conference ID (S102). If the communications unit 51 receives a conference ID attached to the device identification information, the communications management unit 54 does not issue a conference ID.

[0194] The equipment management unit 55 associates the conference ID with the received equipment identification information and stores it in the association information storage unit 5003 (S103). The equipment management unit 55 maintains the association between the conference ID and the equipment identification information until the end of the conference (end of recording).

[0195] The communication unit 51 of the information processing system 50 transmits the conference ID to the terminal device 10 and the electronic whiteboard 2 (S104).

[0196] Figure 22 is a flowchart illustrating the operation of the information recording application 41 running on the terminal device 10. This process is performed because if the information recording application 41 cannot detect the two-dimensional code due to a person standing in front of the electronic whiteboard 2, the electronic whiteboard 2 cannot be associated with the meeting.

[0197] First, the information recording application 41 is launched (S111).

[0198] Next, the time measurement unit 25 starts measuring the time from when the information recording application 41 is launched until the two-dimensional code is detected (S112).

[0199] The device communication unit 16 receives a panoramic image, and the code analysis unit 22 detects the two-dimensional code. The time measurement unit 25 determines whether or not the two-dimensional code was detected within a certain time after startup (S113).

[0200] If a two-dimensional code is detected within a certain period of time after startup (Yes in S113), the time measurement unit 25 stops measuring time (S114).

[0201] If a two-dimensional code is not detected within a certain period of time after startup (No. in S113), the display control unit 13 displays an error dialog in response to a notification from the time measurement unit 25 (S115).

[0202] Figure 23 shows an example of a message 301 displayed in an error dialog by the information recording application 41 of the terminal device 10. This error dialog displays a message 301 such as, "The QR code was not detected. Please ensure that the QR code is displayed and that there is no one in front of the electronic whiteboard 2." The user can understand from the message 301 that the QR code was not detected and take appropriate action, such as moving people away from the electronic whiteboard 2. The OK button 302 is a button that allows the information recording application 41 to measure time again.

[0203] Returning to Figure 22, if the OK button is pressed in step S115 and the error dialog is closed (Yes in S116), the process returns to step S112, and the time measurement unit 25 starts measuring time.

[0204] Furthermore, when a user closes an error dialog, a cancel button may be added to the error dialog, allowing the meeting to be started without associating it with the electronic whiteboard 2.

[0205] <Two-dimensional codes and barcodes displayed on the interactive whiteboard> Refer to Figures 24 to 31 to supplement the information regarding the two-dimensional codes and barcodes displayed by the electronic whiteboard 2. If there is an obstacle 69 between the meeting device 60 and the electronic whiteboard 2, the terminal device 10 may not be able to acquire the device identification information. For this reason, it is effective to adjust the position in which the electronic whiteboard 2 displays the two-dimensional code 8, as described below.

[0206] Figure 24 shows an example of the position of the two-dimensional code 8 displayed by the interactive whiteboard 2. As described above, the two-dimensional code 8 contains device identification information and a statement indicating that it is device identification information. On the desk where the meeting device 60 is placed, there is an obstacle 69 between the meeting device 60 and the interactive whiteboard 2. Note that the meeting device 60 does not need to be a hemispherical camera for the detection of the two-dimensional code 8. The interactive whiteboard 2 displays the device identification information above the center line 320 of the display screen of the interactive whiteboard 2 in the vertical direction. This makes it easier for the meeting device 60 to capture the device identification information even if there is an obstacle 69 between the meeting device 60 and the interactive whiteboard 2.

[0207] As shown in Figure 25, the electronic whiteboard 2 may display the center of the two-dimensional code 8 above the center line 320 of the display screen of the electronic whiteboard 2 in the vertical direction.

[0208] Furthermore, as shown in Figure 26, the electronic whiteboard 2 may move the two-dimensional code 8 over time. In Figure 26, the two-dimensional code 8 moves from left to right. The electronic whiteboard 2 may move while displaying the two-dimensional code 8, or it may display, erase, change location, and redisplay. This makes it easier for the meeting device 60 to capture device identification information even if the position of the obstacle 69 is uncertain. Also, even if the electronic whiteboard 2 displays the two-dimensional code 8 in a smaller size to reduce the feeling of pressure on the user, it still makes it easier for the meeting device 60 to capture it.

[0209] Furthermore, the electronic whiteboard 2 may change the size of the two-dimensional code 8 while moving it.

[0210] Furthermore, as shown in Figure 27, the electronic whiteboard 2 may display multiple two-dimensional codes 8 simultaneously. This makes it easier for the meeting device 60 to capture the remaining two-dimensional codes 8 even if some of them are hidden by an obstacle 69. Note that the multiple two-dimensional codes 8 may all contain the same information, or they may contain different information.

[0211] Furthermore, as shown in Figure 28, the electronic whiteboard 2 may display a two-dimensional code 8 next to (in close proximity to) the menu 71. The menu 71 is positioned vertically along the right edge, and is displayed above the center line of the display screen of the electronic whiteboard 2 in the vertical direction, as in Figure 24.

[0212] By displaying QR code 8 near menu 71, users are less likely to feel any discomfort. Users can also utilize the screen more effectively.

[0213] Furthermore, as shown in Figure 29, the electronic whiteboard 2 may display a two-dimensional code 8 within the menu 71. Compared to Figure 28, this further reduces the feeling of being cramped and allows the user to use the screen more effectively.

[0214] Figure 30 illustrates the case where the camera of the meeting device 60 is a hemispherical camera. By imaging the configurations shown in Figures 24 to 29 with a hemispherical camera that has a wide horizontal field of view, the detection of the two-dimensional code 8 can be made easier.

[0215] Note that barcodes, rather than 2D codes (8), can be displayed in the same way as in Figures 24-30. Figure 31 is an example of a barcode (7).

[0216] Furthermore, barcode 7 is less robust to tilt and other factors than two-dimensional code 8. Therefore, the code analysis unit 22 cuts out the black and white pattern and corrects the skew angle and pitch angle. The code analysis unit 22 enhances the edges of the black bars. The code analysis unit 22 performs pattern matching on the cut-out image with the pattern from the start character to the rightmost stop character that is registered as the pattern of barcode 7, and detects the barcode 7 displayed on the electronic whiteboard 2.

[0217] Furthermore, while Figures 24 to 31 illustrate examples where a two-dimensional code 8 or barcode 7 is displayed, the electronic whiteboard 2 may display device identification information (alphabetical characters or numbers), and the information recording application 41 may detect the device identification information using OCR processing.

[0218] <<Saving combined images and videos>> Next, the process of saving the combined image video will be explained with reference to Figure 32. Figure 32 is an example of a sequence diagram showing the procedure by which the information recording application 41 records a panoramic image, speaker images, and the application screen. As shown in Figure 20, the conference ID has been sent to the terminal device 10 and the electronic whiteboard 2, so the information recording application 41 can record the recording information.

[0219] S21: The user starts a remote conference by operating the remote conference application 42. Here, it is assumed that the remote conference applications 42 at the local site 102 and the other site 101 have started a remote conference. The remote conference application 42 at local site 102 transmits the image captured by the camera and the audio collected by the microphone of the meeting device 60 to the remote conference application 42 at the other site 101. The remote conference application 42 at the other site 101 displays the received image on its display and outputs the received audio through its speaker. Similarly, the remote conference application 42 at the other site 101 transmits the image captured by the camera and the audio collected by the microphone of the meeting device 60 to the remote conference application 42 at local site 102. The remote conference application 42 at local site 102 displays the received image on its display and outputs the received audio through its speaker. Each remote conference application 42 repeats this process to realize a remote conference.

[0220] S22: The user configures the recording settings on the recording settings screen 210 of the information recording application 41 shown in Figure 17. The operation reception unit 12 of the information recording application 41 accepts the settings. Here, it is assumed that both the camera toggle button 211 and the PC screen toggle button 212 are turned on.

[0221] If a user has already scheduled a remote meeting, they can display a list of remote meetings by pressing the information acquisition button 221 on the calendar in Figure 18 and select the remote meeting to which they want to associate a recorded video. Since the user is logged into the information processing system 50, the information processing system 50 identifies the remote meetings that the logged-in user has access to. The information processing system 50 sends the list of identified remote meetings to the terminal device 10, and the user selects a remote meeting that is currently in progress or will be held soon. This determines information about the remote meeting, such as the meeting ID. In this embodiment, the meeting ID has already been created in the process shown in Figure 20.

[0222] S23: The user instructs the information recording application 41 to start recording (by pressing the "Start Recording Now" button 216). The operation reception unit 12 of the information recording application 41 receives the instruction. The display control unit 13 displays the recording screen 220.

[0223] S24: Since the meeting ID is fixed, the communication unit 11 of the information recording application 41 specifies the meeting ID and requests the information processing system 50 for the storage location information.

[0224] S25: The communication unit 51 of the information processing system 50 receives the request, and the communication management unit 54 sends the destination of the combined image video (URL of the storage service system 70) to the information recording application 41 via the communication unit 51.

[0225] S26: The communication unit 11 of the information recording application 41 receives the conference ID and the location where the recording file will be saved, causing the recording control unit 17 to determine that the recording is ready and to start recording.

[0226] S27: The application screen acquisition unit 14 of the information recording application 41 requests the application screen selected by the user (more specifically, the application screen acquisition unit 14 acquires the application screen via the OS). In Figure 32, the application selected by the user is the remote conferencing application 42.

[0227] S28: The recording control unit 17 of the information recording application 41 notifies the meeting device 60 to start recording via the device communication unit 16. When notifying, the recording control unit 17 may also notify that the camera toggle button 211 is on (request for panoramic image and speaker image). Regardless of whether a request is made, the meeting device 60 sends the panoramic image and speaker image to the information recording application 41.

[0228] S29: When the terminal communication unit 61 of the meeting device 60 receives a recording start notification, it assigns a unique recording ID and returns the recording ID to the information recording application 41. The recording ID may be assigned by the information recording application 41 or obtained from the information processing system 50.

[0229] S30: The audio acquisition unit 15 of the information recording application 41 acquires the audio data output by the terminal device 10 (audio data received by the remote conferencing application 42).

[0230] S31: The device communication unit 16 transmits the voice data acquired by the voice acquisition unit 15 and the synthesis request to the meeting device 60.

[0231] S32: The terminal communication unit 61 of the meeting device 60 receives the audio data and synthesis request, and the speech synthesis unit 65 synthesizes the received audio data with the surrounding audio data collected by the sound collection unit 64. For example, the speech synthesis unit 65 adds the two audio data together. Since clear audio from the vicinity of the meeting device 60 is recorded, the accuracy of text transcription of audio, especially from the vicinity of the meeting device 60 (towards the conference room), is improved.

[0232] This voice synthesis can also be performed on the terminal device 10. The recording function may be distributed to the meeting device 60, and the voice processing to the terminal device 10. In this case, the load on the meeting device 60 can be reduced.

[0233] S33: In addition, the panoramic image creation unit 62 of the meeting device 60 creates a panoramic image, and the speaker image creation unit 63 creates a speaker image.

[0234] S34: The device communication unit 16 of the information recording application 41 repeatedly acquires panoramic images and speaker images from the meeting device 60. The device communication unit 16 also repeatedly acquires synthesized audio data from the meeting device 60. These acquisitions may be performed by the device communication unit 16 requesting them from the meeting device 60. Alternatively, the meeting device 60 may automatically send the panoramic images and speaker images upon receiving a request to synthesize audio data. The meeting device 60 may automatically send the synthesized audio data to the information recording application 41 upon receiving a request to synthesize audio data.

[0235] S35: The recording control unit 17 of the information recording application 41 creates a combined image by arranging the application screen acquired from the remote conferencing application 42, a panoramic image, and a speaker image. The recording control unit 17 repeatedly creates combined images and creates a combined image video by specifying each combined image as a frame that makes up the video. The recording control unit 17 also saves the audio data received from the meeting device 60.

[0236] S36: The communication unit 36 ​​of the electronic whiteboard 2 preferably transmits object information such as handwritten data to the information processing system 50, associating it with the conference ID for each stroke.

[0237] The information recording application 41 repeats the above steps S30 to S36.

[0238] Note that the processing in steps S30 to S36 does not have to be in the order shown in Figure 32, and the order in which the audio data is synthesized and the combined image is created may also be different.

[0239] <Main effects> As described above, in this embodiment, the recording information creation system 100 allows the meeting device 60 to acquire equipment identification information of equipment used in the conference room, and the terminal device 10 to transmit this information to the information processing system 50, thereby enabling the information processing system 50 to associate the equipment with equipment used in the conference. Since the meeting device 60 is connected to the terminal device 10 that communicates with the information processing system 50, the meeting device 60 is also associated with the conference. There is no need for the user to capture a two-dimensional code displayed by the equipment with a camera or to register the equipment and the meeting device 60 with the information processing system 50 for the conference, and multiple devices (meeting device 60, electronic whiteboard) can be associated with minimal user effort. [Examples]

[0240] This embodiment describes an electronic whiteboard 2 that notifies device identification information by sound and its processing. This embodiment assumes that the hardware configuration diagrams shown in Figures 4, 5, and 8 described in the above embodiment can be used as reference.

[0241] <About the features> Figure 33 is an example of a functional block diagram that explains the functions of the terminal device 10, meeting device 60, and information processing system 50 in the recording information creation system 100 by dividing them into blocks. The explanation of Figure 33 mainly focuses on the differences from Figure 9.

[0242] In Figure 33, the terminal device 10 has a sound data analysis unit 23. As will be described later, the sound data analysis unit 23 performs spectral analysis on the sound data to detect frequencies contained in the sound data and obtains device identification information by converting specific frequencies into bit data.

[0243] Figure 34 is a functional block diagram illustrating the functions of the electronic whiteboard 2, divided into blocks. This explanation of Figure 34 primarily focuses on the differences from Figure 13. The electronic whiteboard 2 in Figure 34 has a sound data generation unit 37. The sound data generation unit 37 converts the alphanumeric characters that constitute the device identification information into frequencies at time T, and generates a sound signal by sampling them at regular intervals, similar to PCM conversion. The sound signal is converted to analog by the D / A converter in the speaker 450 and output from the speaker 450. The sound data generation unit 37 also serves as an output unit for outputting the sound signal.

[0244] Furthermore, in this embodiment, the sound collection unit 64 of the meeting device 60 also serves as the sound data acquisition unit.

[0245] <Generating audio data> Figure 35 shows an example of the sound frequencies and bit patterns used to transmit device identification information. In Figure 35, the frequencies used to transmit device identification information by sound are 18kHz, 19kHz, and 20kHz. 18kHz is the frequency of the pilot signal added before the message. The length of time T corresponds to 1 bit. Also, 19kHz corresponds to 0 and 20kHz corresponds to 1. Therefore, Figure 35(a) shows the value 00, Figure 35(b) shows the value 01, Figure 35(c) shows the value 10, and Figure 35(d) shows the value 11.

[0246] The pilot signal is transmitted with an additional 2 bits for every 8 bits of data. If the sound signal represents one letter or number with 8 bits, then 10 bits are required to transmit one character of device identification information. If the electronic whiteboard 2 transmits, for example, the 8-bit ASCII character "e" (01100101 in binary: 0x65 in hexadecimal), the frequency pattern of the sound signal will be as shown in Figure 36.

[0247] The device identification information of the electronic whiteboard 2 uses the digits 0-9 and uppercase English letters A-Z. Therefore, the device identification information is represented by an 8-bit ASCII code. Each character of the device identification information, along with a pilot signal, has a frequency pattern as shown in Figure 36. The electronic whiteboard 2 has pre-stored PCM-encoded audio signals corresponding to the frequency patterns of the digits 0-9 and letters A-Z, and a time T of 1 bit. The sound data generation unit 37 converts the PCM-encoded audio data for each character constituting the device identification information of the electronic whiteboard 2 into digital audio data using a sound source.

[0248] The sound collection unit 64 of the meeting device 60 collects ambient sounds, and therefore also collects the sound signals generated by the sound data generation unit 37. The sound data analysis unit 23 of the terminal device 10 performs spectral analysis (Fourier transform) on the sound signals at regular intervals (e.g., several tens of milliseconds) to obtain a spectrum with peaks at 18kHz, 19kHz, or 20kHz. The sound data analysis unit 23 detects the beginning of the characters constituting the device identification information at 18kHz and converts 19kHz or 20kHz to 0 or 1. To clarify, since the time used for spectral analysis (e.g., several tens of milliseconds) is shorter than time T, the sound data analysis unit 23 aggregates the data into 1 bit of 0 or 1 per time T, depending on whether 0 or 1 is more frequent at each time T, and reproduces the device identification information. The time used for spectral analysis (e.g., several tens of milliseconds) and time T may be the same.

[0249] <Operating Procedure> Figure 37 is a sequence diagram illustrating the process performed by the recording information creation system 100. The explanation of Figure 37 mainly focuses on the differences from Figure 20. First, step S41 can be the same as step S1 in Figure 20.

[0250] S42: The user presses a button to output the device identification information of the electronic whiteboard 2, which is placed in the conference room used for meetings. The sound data generation unit 37 of the electronic whiteboard 2 generates an audio signal that represents its own device identification information in terms of frequency and outputs it from the speaker. The audio signal may be output automatically when the power is turned on.

[0251] S43: The sound collection unit 64 of the meeting device 60 collects the sound signal output by the electronic whiteboard 2 using a microphone and encodes the sound signal using PCM encoding.

[0252] S44: The terminal communication unit 61 of the meeting device 60 transmits an audio signal to the terminal device 10.

[0253] S45: The device communication unit 16 of the terminal device 10 receives an audio signal, and the audio data analysis unit 23 performs frequency analysis on the audio signal, divides it for each 18 kHz pilot signal, and converts the frequencies (19 kHz, 20 kHz) included in each divided audio signal into an 8-bit bit string based on the conversion rule in FIG. 35. The audio data analysis unit 23 converts it into device identification information using an 8-bit ASCII code conversion table. This conversion table is a table that associates the numbers 0 to 9 and the uppercase English letters A to Z (8-bit bit strings) with the frequency patterns shown in FIG. 36.

[0254] S46: The communication unit 11 of the terminal device 10 transmits the device identification information to the information processing system 50. The subsequent processing may be the same as that in FIG. 20.

[0255] In FIG. 37, the meeting device 60 collects the audio signal, but the terminal device 10 may directly collect the audio signal. <明

[0256] Also, in the above embodiment, the frequency of the sound is set to 18 kHz to 20 kHz. However, since this frequency is in the audible range, there is a possibility that the user may hear the sound. Therefore, for example, the electronic blackboard 2 may output the device identification information as sound using ultrasonic waves of about 50 to 100 kHz.

[0257] FIG. 38 shows an example of the correspondence between the frequency and bit data in the case of ultrasonic waves. In FIG. 38, the frequencies of 50 kHz (using a pilot signal), 55 kHz, and 60 kHz are shown as ultrasonic waves, but the ultrasonic waves may be other frequencies. FIG. 38(a) shows a value of 00, FIG. 38(b) shows a value of 01, FIG. 38(c) shows a value of 10, and FIG. 38(d) shows a value of 11.

[0258] In this case, it is preferable that the speaker 450 of the electronic blackboard 2 is also compatible with ultrasonic waves and the microphone 608 of the meeting device 60 is also compatible with ultrasonic waves.

[0259] <Main effects> According to this embodiment, in addition to the effects of Embodiment 1, the electronic whiteboard 2 can notify the meeting device 60 of device identification information by sound, making it easier for the meeting device 60 to acquire device identification information even if there is a person in front of the electronic whiteboard 2. [Examples]

[0260] This embodiment describes the processing that occurs when recording ends (meeting ends). When the user ends the recording, the following actions are performed: the association between the meeting ID and the device identification information is deleted in the information processing system 50 (the electronic whiteboard 2 is released), the recording end is displayed on the electronic whiteboard 2, and the display and output of the device identification information is resumed.

[0261] In this embodiment, we will use the hardware configuration diagrams in Figures 4, 5, and 8, and the functional block diagram in Figure 9, which were described in the above embodiment, as references.

[0262] <Action or process> Figure 39 is a sequence diagram illustrating the process by which the recording information creation system 100 terminates recording of a meeting.

[0263] S51: The user is about to end a meeting in which the meeting device 60 and the electronic whiteboard 2 were used in the same meeting. The user presses the recording end button 227 in the information recording application 41. The operation reception unit 12 receives the press.

[0264] S52: The recording control unit 17 of the information recording application 41 stops recording the combined image video and stops recording the audio data.

[0265] S53: In addition, the communication unit 11 of the information recording application 41 sends a meeting completion notification to the information processing system 50 by specifying the meeting ID.

[0266] S54: The communications unit 51 of the information processing system 50 receives a meeting end notification, and the communication management unit 54 sends the meeting end notification to the electronic whiteboard 2 that has transmitted the meeting ID.

[0267] S55: The communication unit 36 ​​of the electronic whiteboard 2 receives a meeting end notification, and the display control unit 34 displays the meeting end notification screen.

[0268] Figure 40 shows an example of a meeting end notification screen 310 displayed by the electronic whiteboard 2. The meeting end notification screen 310 includes a message 311 that reads, "The meeting (recording of information) is now over. Any further handwritten entries will not be saved in the record. If you wish to end the meeting, please press End; if you wish to continue, please press Continue," an End button 312 (an example of a display component), and a Continue button 313. When the user presses the End button 312, the recording of objects displayed on the electronic whiteboard 2 ends (object information is not sent to the information processing system 50). When the Continue button 313 is pressed, the user can add to the electronic whiteboard 2.

[0269] S56: The user pressed the exit button 312. The contact position detection unit 31 detected the press.

[0270] S57: The communication unit 36 ​​of the electronic whiteboard 2 sends a meeting completion confirmation notice to the information processing system 50, specifying the meeting ID. The communication unit 36 ​​then terminates the transmission of object information to the information processing system 50.

[0271] S58: When the communications unit 51 of the information processing system 50 receives a meeting termination confirmation notice, the equipment management unit 55 deletes the association between the meeting ID and the equipment identification information (association information).

[0272] S59: The communication unit 51 of the information processing system 50 sends a notification of completion of disconnection to the electronic whiteboard 2.

[0273] S60: The communication unit 36 ​​of the electronic whiteboard 2 receives a notification that the connection has been terminated and resumes outputting a two-dimensional code or barcode in the case of Example 1, or an audio signal in the case of Example 2. The data recording unit 33 deletes the conference ID.

[0274] S61: The communication unit 51 of the information processing system 50 transmits a notification of completion of cooperation cancellation to the terminal device 10. The communication unit 11 of the information recording application 41 receives the notification of completion of cooperation cancellation and deletes the conference ID.

[0275] S62: In response to the notification of completion of cooperation cancellation, the device communication unit 16 of the information recording application 41 notifies the meeting device 60 of the end of recording. The meeting device 60 continues to create panoramic images, speaker images, and synthesize voices. However, the meeting device 60 may change the processing such as changing the resolution and frame rate depending on whether it is recording. The meeting device 60 may interrupt the creation of panoramic images, speaker images, and voice synthesis due to inactivity of the information recording application 41 or the like.

[0276] S63: The recording control unit 17 of the information recording application 41 combines audio data with the combined image video to create a combined image video with audio.

[0277] S64: Also, when the user checks the checkbox 215 associated with "Automatically perform character recognition after uploading the recording" on the recording setting screen 210, the audio data processing unit 18 requests the information processing system 50 to convert the audio data into text data. Specifically, the audio data processing unit 18 specifies the URL of the storage destination via the communication unit 11 and transmits a conversion request for the audio data combined with the combined image video to the information processing system 50 together with the conference ID and the recording ID.

[0278] S65: The communication unit 51 of the information processing system 50 receives the conversion request for the audio data, and the text conversion unit 56 uses the speech recognition service system 80 to convert the audio data into text data. The communication unit 51 stores the text data in the same storage destination as the storage destination of the combined image video. Note that in the recording information storage unit 5002, the text data is associated with the combined image video by the conference ID ID of the conference ID and the recording ID. The terminal device 10 may request speech recognition from the speech recognition service system 80 and store the text data obtained from the speech recognition service system 80 in the storage destination.

[0279] S66: In addition, the upload unit 20 of the information recording application 41 saves the combined image video to the combined image video storage location via the communication unit 11. In the recording information storage unit 5002, the combined image video is associated with the conference ID and the recording ID. Uploaded combined image video is recorded.

[0280] S67: Furthermore, the communication unit 51 of the information processing system 50 associates object information transmitted from the electronic whiteboard 2 during the meeting with the meeting ID and saves it to the same storage location as the combined image video. Therefore, object information, combined image video, and text data are associated with the meeting ID.

[0281] Since the save location is notified to the user, the user can share the combined image / video with participants by informing them of the save location via email or other means. Even if the devices used to create the combined image / video, audio data, text data, and object information are different, they can be collected and stored in a single location, making them easily viewable by users later.

[0282] <Main effects> According to this embodiment, upon termination of recording (end of meeting), it becomes possible to delete the association between the meeting ID and device identification information in the information processing system 50 (release of the electronic whiteboard 2), display the end of recording on the electronic whiteboard 2, and resume the display and output of device identification information. [Examples]

[0283] In this embodiment, the processing at the end of recording will be described in the same way as in Embodiment 3, but a different termination method will be described.

[0284] In this embodiment, we will use the hardware configuration diagrams shown in Figures 4, 5, and 8 described in the above embodiment as a reference.

[0285] <About the features> Figure 41 is an example of a functional block diagram that explains the functions of the terminal device 10, meeting device 60, and information processing system 50 in the recording information creation system 100 by dividing them into blocks. The explanation of Figure 41 mainly focuses on the differences from Figure 9.

[0286] In Figure 41, the terminal device 10 has an end detection unit 24. The end detection unit 24 stores the shape of the electronic whiteboard 2 and determines whether the electronic whiteboard 2 can be detected from the panoramic image. First, when a two-dimensional code or barcode is detected, the end detection unit 24 uses pattern matching or machine learning to detect the shape (circumscribed rectangle) of the electronic whiteboard 2 in the conference room.

[0287] Figure 42 is an image 241 of the shape of the electronic whiteboard 2 detected from the panoramic image. During recording, the termination detection unit 24 repeatedly detects the electronic whiteboard 2 from the panoramic image using Figure 42 as the standard pattern, and determines that the meeting (recording) has ended when it determines that the electronic whiteboard 2 is not present.

[0288] This process allows the meeting to be ended without the user having to press the recording stop button 227. Alternatively, the user can end the meeting by blocking the space between the meeting device 60 and the electronic whiteboard 2 with their hand (or by using the mute button). Furthermore, the user can end the meeting by leaving the meeting room while the terminal device 10 and the meeting device 60 are still connected.

[0289] Furthermore, the termination detection unit 24 may detect the end of the meeting by detecting the unplugging of the USB cable (wired cable). The device communication unit 16 detects that the USB cable has been unplugged, for example, when the external device connection I / F 508 does not detect voltage, and notifies the termination detection unit 24. The device communication unit 16 may also detect a communication interruption due to no response from the meeting device 60, etc. In this case as well, the meeting can be ended by the user unplugging the cable, a normal operation, without having to press the recording end button 227.

[0290] <Action or process> Figure 43 is a sequence diagram illustrating the process by which the recording information creation system 100 completes the recording of a meeting. The explanation of Figure 43 mainly focuses on the differences from Figure 39.

[0291] S70: The terminal communication unit 61 of the meeting device 60 transmits a panoramic image to the terminal device 10.

[0292] S71: The termination detection unit 24 detects that the electronic whiteboard 2 is not detected in the panoramic image as described above, or that the USB cable has been unplugged. Note that the meeting device 60 can also detect that the electronic whiteboard 2 is not detected in the panoramic image.

[0293] The subsequent steps can be the same as in Figure 39.

[0294] <Main effects> According to this embodiment, in addition to the effects of Embodiment 3, the user's operational steps can be reduced, and the meeting (recording) can be terminated. [Examples]

[0295] In this embodiment, the processing at the end of recording will be described in the same way as in Embodiment 3, but a different termination method will be described.

[0296] In this embodiment, we will use the hardware configuration diagrams shown in Figures 4, 5, and 8 described in the above embodiment as a reference.

[0297] <About the features> Figure 44 is an example of a functional block diagram that explains the functions of the terminal device 10, meeting device 60, and information processing system 50 in the recording information creation system 100 by dividing them into blocks. The explanation of Figure 44 mainly focuses on the differences from Figure 33.

[0298] In Figure 44, the terminal device 10 has an end detection unit 24. The end detection unit 24 detects that the meeting has ended when it determines that the pilot signal is not included in the sound captured by the microphone of the meeting device 60. In other words, during the meeting (recording), the electronic whiteboard 2 intermittently outputs a pilot signal, and the end of the meeting (end of recording) can be detected when the meeting device 60 stops receiving the pilot signal.

[0299] With this process, the meeting can be ended by either blocking the space between the meeting device 60 and the electronic whiteboard 2 with one's hand, or by leaving the meeting room while the terminal device 10 and the meeting device 60 are still connected, without the user having to press the recording stop button 227.

[0300] <Action or process> Figure 45 is a sequence diagram illustrating the process by which the recording information creation system 100 completes the recording of a meeting. The explanation of Figure 45 mainly focuses on the differences from Figure 39.

[0301] S80: The terminal communication unit 61 of the meeting device 60 transmits sound data to the terminal device 10.

[0302] S81: The termination detection unit 24 determines whether or not the sound data transmitted from the meeting device 60 includes a pilot signal, as described above. This determination may be made directly by the information recording application 41 using the sound collected, or by receiving a result of no detection from the meeting device 60.

[0303] The subsequent steps can be the same as in Figure 39.

[0304] <Main effects> According to this embodiment, in addition to the effects of Embodiment 3, the user's operational steps can be reduced, and the meeting (recording) can be terminated.

[0305] <Other application examples> Although the best mode for carrying out the present invention has been described above using examples, the present invention is not limited in any way to these examples, and various modifications and substitutions can be made without departing from the spirit of the present invention.

[0306] For example, the terminal device 10 and the meeting device 60 may be integrated into one unit. Alternatively, the meeting device 60 may be attached externally to the terminal device 10. Furthermore, the meeting device 60 may consist of a hemispherical camera, a microphone, and a speaker connected by cables.

[0307] Meeting devices 60 may also be placed at other locations 101. These other locations 101 will separately use meeting devices 60 to create combined image / video and text data. Furthermore, multiple meeting devices 60 may be placed at a single location. In this case, multiple recordings will be created for each meeting device 60.

[0308] Furthermore, the arrangement of the panoramic image 203, speaker image 204, and application screen in the combined image video used in this embodiment is merely an example. The panoramic image 203 may be at the bottom and the speaker image 204 at the top, or the user may change the arrangement, or individually switch the display of the panoramic image 203 and speaker image 204 during playback.

[0309] Furthermore, the configuration examples shown in Figure 9 and other figures are divided according to their main functions to facilitate understanding of the processing performed by the terminal device 10, meeting device 60, and information processing system 50. The present invention is not limited by the way the processing units are divided or the names of those units. The processing of the terminal device 10, meeting device 60, and information processing system 50 can be further divided into more processing units depending on the processing content. Alternatively, a single processing unit can be divided to include even more processing.

[0310] Furthermore, the apparatus described in the examples represents only one of several computing environments for carrying out the embodiments disclosed herein. In one embodiment, the information processing system 50 includes multiple computing devices, such as a server cluster. The multiple computing devices are configured to communicate with each other via any type of communication link, including networks and shared memory, and perform the processing disclosed herein.

[0311] Furthermore, the information processing system 50 can be configured to share the disclosed processing steps, such as those shown in Figure 20, in various combinations. For example, a process executed by a predetermined unit can be executed by multiple information processing devices of the information processing system 50. Also, the information processing system 50 may be consolidated into a single server device or divided into multiple devices.

[0312] Each of the embodiments described above can be implemented by one or more processing circuits. Here, "processing circuit" as used herein includes processors programmed to execute each function by software, such as processors implemented by electronic circuits, as well as devices such as ASICs (Application Specific Integrated Circuits), DSPs (Digital Signal Processors), FPGAs (Field Programmable Gate Arrays), and conventional circuit modules designed to execute each of the functions described above. [Explanation of symbols]

[0313] 10 Terminal devices 50 Information Processing Systems 60 Meeting Devices 100 Record Information Creation System [Prior art documents] [Patent Documents]

[0314] [Patent Document 1] Japanese Patent Publication No. 2008-124689

Claims

1. A device management system having multiple devices and an information processing system for processing information related to communication, The first device is, It has an output unit that outputs device identification information of the first device, A second device capable of communicating with the terminal device is, An acquisition unit that acquires the device identification information output by the first device, It includes a terminal communication unit that transmits the aforementioned device identification information to the terminal device, The aforementioned information processing system is A communication unit that receives the device identification information from the terminal device, A device management unit that, upon receiving the aforementioned device identification information, makes the first device available for communication, A device management system characterized by having the following features.

2. The system includes a communication management unit that issues communication identification information in response to the reception of the aforementioned device identification information. The device management system according to claim 1, characterized in that the communication unit transmits the communication identification information to the first device and the terminal device.

3. The device management system according to claim 2, characterized in that the device management unit makes the first device available for communication by associating the device identification information and the identification information of the first device transmitted from the terminal device with the communication identification information and storing them together.

4. The first device transmits the object information displayed by the first device to the information processing system in association with the communication identification information. The information processing system receives ambient image data and communication identification information from the terminal device, which the terminal device has received from the second device. The device management system according to any one of claims 1 to 3, characterized in that the information processing system stores the object information and the surrounding image data in association with the communication identification information.

5. The output unit displays a two-dimensional code or barcode on which the device identification information is encoded. The acquisition unit captures the two-dimensional code or the barcode, The device management system according to any one of claims 1 to 4, characterized in that the communication unit receives the device identification information decoded from the two-dimensional code or the barcode from the terminal device.

6. The device management system according to claim 5, characterized in that the output unit displays the two-dimensional code or the barcode above the center line of the display screen of the first device.

7. The device management system according to claim 5, characterized in that the output unit displays the center of the two-dimensional code or the barcode above the center line of the display screen of the first device.

8. The output unit outputs an audio signal in which the device identification information is encoded. The acquisition unit collects the sound signal, The device management system according to any one of claims 1 to 4, characterized in that the communication unit receives the device identification information decoded from the sound signal from the terminal device.

9. The equipment management system according to claim 8, characterized in that the output unit outputs the sound signal transmitted by the frequency of ultrasound.

10. The terminal device has a time measuring unit that measures the time elapsed since the application that communicates with the second device was launched. The device management system according to any one of claims 1 to 9, characterized in that if the time measuring unit measures a certain period of time but the acquisition unit does not acquire the device identification information, it displays a message indicating that the device identification information was not acquired.

11. The terminal device includes a recording control unit that records ambient image data received from the second device, It has an termination detection unit that detects when the recording of the surrounding image data is to be terminated, The termination detection unit transmits a message to the information processing system indicating that it has terminated the recording. The device management system according to claim 4, characterized in that the information processing system notifies the first device that the recording of the object information has been completed.

12. The device management system according to claim 11, characterized in that the termination detection unit detects that recording should be terminated in response to the pressing of the recording termination button displayed by the terminal device.

13. The termination detection unit identifies the shape of the first device based on a two-dimensional code or barcode captured in the surrounding image data received by the terminal device from the second device. The device management system according to claim 11, characterized in that it detects that recording should be terminated if the first device cannot be detected from the surrounding image data repeatedly transmitted from the second device.

14. The second device collects the sound signal output by the first device. The device management system according to claim 11, characterized in that the termination detection unit detects that recording should be terminated if the terminal device cannot detect an audio signal of a predetermined frequency from the audio signal received from the second device.

15. The terminal device and the second device are connected by a wired cable. The device management system according to claim 11, characterized in that the termination detection unit detects that recording should be terminated when the terminal device detects that the wired cable has been unplugged.

16. When the first device receives a notification that the recording of the object information has ended, it displays a display component that accepts whether or not to end the recording of the object information displayed by the first device. The device management system according to any one of claims 11 to 15, characterized in that when the first device receives a request to terminate recording, the first device terminates the transmission of the object information displayed by the first device to the information processing system.

17. An information processing system that communicates with a first device and terminal device that processes information related to communication, A communication unit receives device identification information from the terminal device, which is output by the first device and acquired by a second device capable of communicating with the terminal device. A device management unit that, upon receiving the aforementioned device identification information, makes the first device available for communication, An information processing system characterized by having the following features.

18. A device management method performed by an information processing system that communicates with a first device that processes information related to communication and a terminal device, The communication unit receives from the terminal device the device identification information output by the first device and acquired by the second device which is capable of communicating with the terminal device, The equipment management unit, upon receiving the equipment identification information, makes the first equipment available for communication; A method for managing equipment, characterized by having the following features.

19. A primary device that processes information related to communication, a terminal device, and an information processing system that communicates with them. A communication unit receives device identification information from the terminal device, which is output by the first device and acquired by a second device capable of communicating with the terminal device. In response to receiving the aforementioned device identification information, the device management unit makes the first device available for communication. A program designed to function as such.

20. A device that processes information related to communication, An acquisition unit that acquires device identification information output from other devices used for communication, A terminal communication unit transmits the device identification information to the information processing system in order to make the other device available for communication, upon receiving the device identification information. A device characterized by having the following features.

21. A device that can communicate with terminal devices and processes information related to communication, An acquisition unit that acquires device identification information output from other devices used for communication, A terminal communication unit that, upon receiving the device identification information from the terminal device, transmits the device identification information to the terminal device in order to make the other device available for communication, A device characterized by having the following features.

Images