Equipment management system, information processing system, equipment management method, program, equipment

Abstract

To provide an apparatus management system that allows an apparatus that processes information about communication to be used for communication.SOLUTION: The present invention is an apparatus management system 100 that includes a plurality of apparatuses that process information about communication and an information processing system. A first apparatus 2 includes an output unit that outputs apparatus identification information on the first apparatus. A second apparatus 60 capable of communicating with a terminal apparatus 10 includes an acquisition unit that acquires the apparatus identification information output by the first apparatus, and a terminal communication unit that transmits the apparatus identification information to the terminal apparatus. The information processing system includes a communication unit that receives the apparatus identification information from the terminal apparatus, and an apparatus management unit that makes the first apparatus available for communication in response to receiving the apparatus identification information.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a device management system, an information processing system, a device management method, a program, and a device. [Background technology]

[0002] Remote communication systems are known in which users in remote locations can hold conferences by transmitting images and audio in real time from one location to one or more other locations. Remote communication may involve the use of devices such as electronic whiteboards.

[0003] There is known a technology that makes it easy to authenticate participants in a conference (see, for example, Patent Document 1). Patent Document 1 discloses a system that has an imaging means that captures an image of the surroundings of the device and generates video data, and the imaging means reads a participation certificate that has been converted into image data, analyzes the participation certificate that has been converted into image data, and accepts participation in the conference. Summary of the Invention [Problem to be solved by the invention]

[0004] However, conventional technologies have a problem in that devices that process information related to communication must be made available for communication. For example, when an electronic whiteboard is used in a meeting, participants must associate the electronic whiteboard with the meeting in order to save data handwritten on the electronic whiteboard in association with the meeting.

[0005] In view of the above problems, the present invention provides a method for processing information relating to communication. The first device and the second Equipment Match to the same communication The object is to provide a device management system. [Means for solving the problem]

[0006] In view of the above problems, the present invention provides a first device, a second device, and an information processing system for processing information relating to communication. haveAn equipment management system, wherein the first equipment comprises: a display; and an image displayed on the display. an output unit for outputting information, and the second device outputs the image Information By taking pictures an acquisition unit for acquiring the Obtained from image information information In front a terminal communication unit for transmitting the information to the information processing system, Sent by The aforementioned Information obtained from image information a communication unit for receiving the Information obtained from image information and a device management unit that associates the first device and the second device with the same communication based on the communication information. [Effects of the Invention]

[0007] Process information about your communications The first device and the second Equipment Match to the same communication An equipment management system can be provided. [Brief explanation of the drawings]

[0008] [Figure 1] 10A and 10B are diagrams illustrating an outline of creating record information that saves the screen of an application executed during a remote conference together with a panoramic image of the surrounding area. [Figure 2] 10 is a diagram illustrating an outline of a process in which the information processing system associates an electronic whiteboard with a meeting device. FIG. [Figure 3] FIG. 1 is a diagram illustrating an example of the configuration of a record information creation system. [Figure 4] FIG. 2 is a diagram illustrating an example of the hardware configuration of an information processing system and a terminal device. [Figure 5] FIG. 2 is a diagram illustrating an example of the hardware configuration of a meeting device. [Figure 6] FIG. 10 is a diagram illustrating the imaging range of the meeting device. [Figure 7] 10A and 10B are diagrams illustrating the cutting out of a panoramic image and a speaker image. [Figure 8] FIG. 2 is a diagram illustrating a hardware configuration of an example of an electronic whiteboard. [Figure 9] 1 is an example of a functional block diagram illustrating functions of a terminal device, a meeting device, and an information processing system in a record information creation system, divided into blocks. [Figure 10] 4 is a diagram showing an example of moving image recording information stored in an information storage unit. FIG. [Figure 11] FIG. 10 is a diagram illustrating an example of conference information managed by a communication management unit. [Figure 12] 10 is a diagram showing an example of association information stored in an association information storage unit, in which conference IDs are associated with device identification information; FIG. [Figure 13] FIG. 1 is an example of a functional block diagram illustrating functions of an electronic whiteboard divided into blocks. [Figure 14] 4 is a diagram showing an example of device identification information stored in a device information storage unit; FIG. [Figure 15] FIG. 2 is a diagram illustrating object information stored in an object information storage unit. [Figure 16] FIG. 10 is a diagram showing an example of an initial screen displayed by an information recording application running on the terminal device after login. [Figure 17] FIG. 10 is a diagram showing an example of a recording setting screen displayed by an information recording application. [Figure 18] FIG. 10 is a diagram showing an example of a recording screen displayed by the information recording application during recording. [Figure 19] FIG. 10 is a diagram illustrating an example of a conference list screen displayed by the information recording application. [Figure 20] 10 is an example of a sequence diagram illustrating a process in which the record information creation system associates a meeting device with an electronic whiteboard. [Figure 21] 10 is an example of a sequence diagram illustrating a procedure or process in which a user operates an information recording application running on a terminal device to log in to an information processing system. [Figure 22] FIG. 10 is a flowchart illustrating an example of an operation of the information processing system. [Figure 23] FIG. 10 is a diagram showing an example of a message displayed by an information recording application of the terminal device. [Figure 24]10A and 10B are diagrams illustrating an example of a display method for a two-dimensional code displayed on an electronic whiteboard. [Figure 25] 10A and 10B are diagrams illustrating an example of a display method for a two-dimensional code displayed on an electronic whiteboard. [Figure 26] 10A and 10B are diagrams illustrating an example of a display method for a two-dimensional code displayed on an electronic whiteboard. [Figure 27] 10A and 10B are diagrams illustrating an example of a display method for a two-dimensional code displayed on an electronic whiteboard. [Figure 28] 10A and 10B are diagrams illustrating an example of a display method for a two-dimensional code displayed on an electronic whiteboard. [Figure 29] 10A and 10B are diagrams illustrating an example of a display method for a two-dimensional code displayed on an electronic whiteboard. [Figure 30] 10A and 10B are diagrams illustrating an example of a display method for a two-dimensional code displayed on an interactive whiteboard when the camera of the meeting device is a hemispherical camera. [Figure 31] 10A and 10B are diagrams illustrating an example of a display method of a barcode displayed on an electronic whiteboard. [Figure 32] 10 is a sequence diagram illustrating an example of a procedure in which the information recording application records a panoramic image, a speaker image, and an application screen. [Figure 33] 10 is an example of a functional block diagram illustrating functions of a terminal device, a meeting device, and an information processing system in a record information creation system, divided into blocks (Example 2). [Figure 34] FIG. 10 is an example of a functional block diagram for explaining functions of an electronic whiteboard by dividing them into blocks (Example 2); [Figure 35] 10A and 10B are diagrams illustrating an example of sound frequencies and bit patterns used to transmit device identification information. [Figure 36] FIG. 10 is a diagram showing an example of a frequency pattern representing one character of device identification information. [Figure 37] FIG. 10 is an example of a sequence diagram illustrating a process performed by the record information creation system. [Figure 38] FIG. 10 is a diagram showing an example of the correspondence between frequency and bit data in the case of ultrasonic waves. [Figure 39]10 is a sequence diagram illustrating an example of a process performed when the record information creation system ends recording of a conference. [Figure 40] FIG. 10 is a diagram illustrating an example of a meeting end notification screen displayed on the electronic whiteboard. [Figure 41] 10 is an example of a functional block diagram illustrating functions of a terminal device, a meeting device, and an information processing system in a record information creation system, divided into blocks (Example 4). [Figure 42] FIG. 10 is a diagram showing an example of an image of the shape of an electronic whiteboard 2 detected from a panoramic image. [Figure 43] FIG. 10 is a sequence diagram illustrating a process performed when the record information creation system ends recording of a conference (Example 4). [Figure 44] 10 is an example of a functional block diagram illustrating functions of a terminal device, a meeting device, and an information processing system in a record information creation system, divided into blocks (Example 5). [Figure 45] 13 is a sequence diagram illustrating an example of a process performed when the record information creation system ends recording of a conference (Example 5); DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, a device management system and a device management method performed by the device management system will be described as an example of an embodiment of the present invention. [Example]

[0010] <An example of how to create minutes for a remote meeting> First, an outline of a method for creating minutes using a panoramic image and an application screen will be described with reference to Fig. 1. Fig. 1 is a diagram for explaining an outline of creating record information that saves the screen of an application executed during a remote conference together with a panoramic image of the surrounding area. As shown in Fig. 1, a user at a home location 102 is using a remote conference service system 90 to hold a remote conference with another location 101.

[0011] The record information creation system 100 of this embodiment creates record information (minutes) using a horizontal panoramic image (hereinafter referred to as a panoramic image) captured by a meeting device 60 equipped with a microphone and a speaker, and a screen created by an application (hereinafter referred to as an app) executed by the terminal device 10. Regarding audio, the record information creation system 100 synthesizes the audio received by the remote conference app 42 and the audio acquired by the meeting device 60, and includes the synthesized audio in the record information. An outline of this is given below.

[0012] (1) An information recording application 41 and a remote conference application 42, which will be described later, are running on the terminal device 10. In addition, an application for displaying materials may also be running. The information recording application 41 transmits the audio output by the terminal device 10 (including the audio received by the remote conference application 42 from other locations) to a meeting device 60 (an example of a second device). The meeting device 60 mixes (combines) the audio it has acquired with the audio from the remote conference application 42.

[0013] (2) The meeting device 60 is equipped with a microphone and performs processing to extract the speaker from the panoramic image based on the direction from which the voice was captured, thereby creating a speaker image. The meeting device 60 transmits both the panoramic image and the speaker image to the terminal device 10.

[0014] (3) The information recording application 41 running on the terminal device 10 can display the panoramic image 203 and the speaker image 204. The information recording application 41 combines any application screen selected by the user (for example, the screen 103 of the remote conference application 42) with the panoramic image 203 and the speaker image 204. For example, the information recording application 41 combines the panoramic image 203, the speaker image 204, and the application screen 103 so that the panoramic image 203 and the speaker image 204 are on the left and the screen 103 of the remote conference application 42 is on the right (hereinafter referred to as a combined image 105). The process of (3) is executed repeatedly, and the combined image 105 becomes a moving image (hereinafter referred to as a combined image moving image). The information recording application 41 also combines the synthesized audio with the combined image moving image to create a moving image with audio.

[0015] In this embodiment, an example in which the panoramic image 203, the speaker image 204, and the application screen 103 are combined will be described, but the information recording application 41 may store these separately and arrange them on the screen during playback.

[0016] (4) The information recording application 41 accepts editing operations (cutting out unnecessary parts by the user) and completes the combined image video. The combined image video constitutes part of the recorded information.

[0017] (5) The information recording application 41 transmits the created combined image video (with audio) to the storage service system 70 and stores it there.

[0018] (6) The information recording application 41 also extracts only the audio from the combined image video (the audio before combining may be kept), and transmits the extracted audio to the information processing system 50. The information processing system 50 transmits the audio to the speech recognition service system 80, which converts the audio into text data, and converts the audio into text. The text data also includes data on how many minutes after the start of recording that the speech occurred.

[0019] In the case of real-time text conversion, the meeting device 60 directly transmits the voice to the information processing system 50. The information processing system 50 transmits the text data obtained by voice recognition to the information recording application 41 in real time.

[0020] (7) The information processing system 50 additionally stores the text data in the storage service system 70 that stores the combined image video. The text data constitutes part of the recorded information.

[0021] The information processing system 50 can perform billing processing for the user according to the service used. For example, the billing is calculated based on the amount of text data, the file size of the combined image video, the processing time, etc.

[0022] In this way, the combined image video displays a panoramic image of the surroundings including the user and an image of the speaker, and further displays the screen of an app that was displayed during the remote conference, such as the remote conference app 42. When participants in the remote conference or non-participants view the combined image video as minutes, the situation during the remote conference is reproduced with a sense of realism.

[0023] <Processing Overview> Next, the process of associating the electronic whiteboard 2 with the meeting device 60 will be described with reference to Fig. 2. Fig. 2 is a diagram illustrating an outline of the process of the information processing system 50 associating the electronic whiteboard 2 with the meeting device 60.

[0024] (1) A participant brings a meeting device 60 to a conference room where an electronic whiteboard 2 (a first device, an example of another device) is installed and places it on a desk. Therefore, the meeting device 60 is placed near the electronic whiteboard 2.

[0025] (2) The meeting device 60 captures an image of the two-dimensional code in which the device identification information of the interactive whiteboard 2 is embedded, and transmits the image data to the terminal device 10.

[0026] (3) The terminal device 10 detects the device identification information from the image data and transmits it to the information processing system 50.

[0027] (4) Upon receiving the device identification information, the information processing system 50 issues a conference ID and associates the device identification information of the interactive whiteboard 2 with the conference ID.

[0028] The timing at which the information processing system 50 issues a conference ID is not limited to this; for example, the information processing system 50 may issue the conference ID in advance by user operation, or may work in conjunction with a scheduler to automatically issue the conference ID at the start time of the conference.

[0029] (5) The information processing system 50 transmits a registration completion notice including the conference ID to the electronic whiteboard 2 and the terminal device 10. By specifying the conference ID and communicating with the information processing system 50, the electronic whiteboard 2 and the terminal device 10 can associate the data they handle with the same conference. The information processing system 50 records the image data (panoramic image and speaker image) and audio data acquired by the meeting device 60, as well as handwritten data on the electronic whiteboard 2, in association with the conference ID.

[0030] In this way, in the record information creation system 100 of this embodiment, the meeting device 60 acquires device identification information of devices used in a conference room, and the terminal device 10 transmits this information to the information processing system 50, allowing the information processing system 50 to associate the devices as devices used in the conference. Because the meeting device 60 is connected to the terminal device 10 that communicates with the information processing system 50, the meeting device 60 can also be associated with the conference. There is no need for the user to capture the two-dimensional code displayed by the device with a camera or to register the device and meeting device 60 in the information processing system 50 for the conference, and multiple devices (meeting device 60, electronic whiteboard 2) can be associated (made available) with minimal user work. Various examples of "making available" are possible, including making the electronic whiteboard a device used in a meeting, enabling the electronic whiteboard to send handwritten stroke data and images to the information processing system 50, enabling minutes to be created using information from the electronic whiteboard, linking the electronic whiteboard to a meeting, enabling data transmission between the electronic whiteboard and the information processing system 50, and allowing the electronic whiteboard to participate in a cloud electronic whiteboard service (a service in which terminal devices participating in a remote conference share strokes on the same screen). Furthermore, making an electronic whiteboard available for use in a meeting can also prevent the electronic whiteboard from participating in other meetings taking place at the same time. Furthermore, making an electronic whiteboard available for use in a meeting can include display information and writing information (stroke information) from the electronic whiteboard in the minutes of the meeting created by the information processing system, etc.

[0031] <Terminology> An application (app) is software developed and used for a specific function or purpose, and is not intended for the operation of a computer itself (OS: operating system). Applications are divided into native apps and web apps.

[0032] A running app is an app that is in a state between when it is launched and when it is closed. The app does not have to be the active app (the foreground app), it just needs to be running in the background.

[0033] The surrounding image acquired by the meeting device 60 refers to an image captured with a wider horizontal angle of view than the normal angle of view. In this embodiment, the surrounding image is described as a panoramic image. A panoramic image is an image with a horizontal angle of view of approximately 180° to 360°. A panoramic image does not have to be captured by a single meeting device, and multiple imaging devices with normal angles of view may be combined.

[0034] The recorded information is information recorded by the information recording application 41. When the information recording application 41 records the screen of the remote conference application 42, the recorded information may become minutes of the meeting. The recorded information includes, for example, a combined image video (including audio) and text data obtained by voice recognition of the audio.

[0035] A tenant is a group of users (such as a company, a local government, or some of these organizations) who have signed a contract to receive services from a service provider. In this embodiment, the creation of recorded information and conversion to text data are performed because the tenant has signed a contract with the service provider.

[0036] Remote communication refers to communication through audio and video using software and terminal devices with people at physically distant locations. An example of remote communication is a remote conference, where a conference can also be called a meeting, a conference, a get-together, a get-together, or a gathering, and a seminar can also be called a training session, study group, seminar, or workshop.

[0037] A base is a place where activities are based. An example of a base is a conference room. A conference room is a room set up primarily for the purpose of holding meetings.

[0038] Although voice refers to speech sounds uttered by humans, surrounding sounds, etc., and voice data refers to voice converted into data, in the present embodiment, a strict distinction will not be made between them.

[0039] The first device may be any device that displays information. In this embodiment, the first device is described using the term "electronic whiteboard." The electronic whiteboard may also be called an electronic whiteboard, an electronic information board, or the like. A projector is known as a device equivalent to the electronic whiteboard 2. Alternatively, the first device may be a digital signage, a television, a display, a multifunction printer, a video conference terminal, or the like.

[0040] The information related to communication is information recorded during communication such as a meeting, and is, for example, information displayed on an electronic whiteboard, images captured by an imaging device, the voice of a speaker, etc. Examples of devices that process information related to communication are the electronic whiteboard 2 and the meeting device 60. The meeting device 60 may have an imaging device.

[0041] The same identification information transmitted to the first device and the terminal device is information that associates multiple devices in communication. The same identification information may be any information, but in this embodiment, it is described using the term conference ID.

[0042] <System configuration example> Next, the system configuration of the record information creation system 100 will be described with reference to Fig. 3. Fig. 3 shows an example of the configuration of the record information creation system 100. Fig. 3 shows one location (home location 102) among multiple locations where a remote conference is held, and a terminal device 10 at the home location 102 communicates with an information processing system 50, a storage service system 70, and a remote conference service system 90 via a network. A meeting device 60 and an electronic whiteboard 2 are also placed at the home location 102, and the terminal device 10 is communicatively connected to the meeting device 60 via a USB cable or the like. The meeting device 60, the electronic whiteboard 2, and the information processing system 50 operate as a device management system.

[0043] At least an information recording application 41 and a remote conference application 42 run on the terminal device 10. The remote conference application 42 can communicate with terminal devices 10 at other locations 101 via a remote conference service system 90 on the network, allowing users at each location to hold conferences from remote locations. The information recording application 41 creates record information for the remote conferences held by the remote conference application 42 using the functions of the information processing system 50 and the meeting device 60.

[0044] In this embodiment, an example of creating recorded information during a remote conference will be described, but the conference does not have to be a conference involving communication with a remote location. In other words, the conference may be a conference in which only participants from one location participate. In this case, the images captured by the meeting device 60 and the collected audio are saved without being combined, and the processing of the information recording application 41 remains unchanged.

[0045] The terminal device 10 is equipped with a built-in camera with a normal angle of view (or it may be external), which captures a front image including the user 107 operating the terminal device 10. A normal angle of view is an image that is not a panoramic image, and in this embodiment, it is a planar image that is not curved, such as a spherical image. Therefore, the user can hold a conventional teleconference using the teleconference application 42 without being aware of the information recording application 41. The information recording application 41 and the meeting device 60 do not affect the teleconference application 42, except for increasing the processing load on the terminal device 10. The teleconference application 42 can also transmit the panoramic image and speaker images captured by the meeting device 60 to the teleconference service system 90.

[0046] The information recording application 41 communicates with the meeting device 60 to create recorded information. The information recording application 41 also performs tasks such as synthesizing audio captured by the meeting device 60 with audio received from other locations by the remote conference application 42. The meeting device 60 is a meeting device equipped with a panoramic image capture device, a microphone, and a speaker. While the camera of the terminal device 10 can only capture a limited range in front of it, the meeting device 60 can capture the entire surroundings (not necessarily the entire periphery) surrounding the meeting device 60. The meeting device 60 can always fit multiple participants 106 shown in FIG. 3 within its field of view.

[0047] In addition, the meeting device 60 performs operations such as cutting out a speaker image from a panoramic image. The meeting device 60 is not limited to being placed on a desk, but may be placed anywhere in the own base 102. Since the meeting device 60 can capture a celestial sphere image, it may be placed on the ceiling, for example.

[0048] The information recording application 41 displays a list of applications running on the terminal device 10, synthesizes images for the recorded information (creates a combined image video), plays back the combined image video, and accepts editing requests. The information recording application 41 also displays a list of teleconferences that have been held or are scheduled to be held. The list of teleconferences is used for information related to the recorded information, allowing the user to associate the teleconferences with the recorded information.

[0049] The remote conference application 42 performs communication connection with other locations 101, transmission and reception of images and audio with other locations 101, display of images, output of audio, and the like.

[0050] The information recording application 41 and the remote conference application 42 may be web applications or native applications. A web application is an application in which a program on a web server and a program on a web browser cooperate to perform processing, and is an application that does not need to be installed on the terminal device 10. A native application is an application that is installed on the terminal device 10 and used. In this embodiment, both applications will be described as native applications.

[0051] The terminal device 10 may be a general-purpose information processing device with a communication function, such as a PC (Personal Computer), a smartphone, or a tablet terminal. The terminal device 10 may also be an electronic whiteboard 2, a game console, a PDA (Personal Digital Assistant), a wearable PC, a car navigation system, industrial machinery, medical equipment, or network home appliances. The terminal device 10 may be any device on which the information recording application 41 and the remote conference application 42 run.

[0052] The electronic whiteboard 2 displays data handwritten on a touch panel using an input means such as a pen or a finger. The electronic whiteboard 2 can communicate with a terminal device 10 or the like via wired or wireless communication, and can capture screens displayed by the terminal device 10 and display them on the display. The electronic whiteboard 2 can convert handwritten data into text data and share information displayed on the display with electronic whiteboards 2 at other locations. The electronic whiteboard 2 may be a simple whiteboard without a touch panel, in which an image is projected by a projector. The electronic whiteboard 2 may also be a tablet terminal, notebook PC (Personal Computer), PDA (Personal Digital Assistant), game console, or the like, equipped with a touch panel.

[0053] The interactive whiteboard 2 can communicate with the information processing system 50. The interactive whiteboard 2 can receive information from the information processing system 50 by polling the information processing system 50 after being powered on, for example.

[0054] The information processing system 50 is one or more information processing devices arranged on a network. The information processing system 50 has one or more server applications that perform processing in cooperation with the information recording application 41, and a platform service. The server applications manage remote conference lists, manage recorded information recorded in remote conferences, and manage various settings and storage paths. The platform service handles user authentication, contracts, billing, etc.

[0055] All or part of the functions of the information processing system 50 may exist in a cloud environment or an on-premise environment. The information processing system 50 may be configured with multiple server devices or may be configured with a single information processing device. For example, the server application and the infrastructure service may be provided by separate information processing devices, and each function within the server application may have its own information processing device. The information processing system 50 may be integrated with the storage service system 70 and the voice recognition service system 80 described below.

[0056] The storage service system 70 is a storage means on a network and provides a storage service that accepts the saving of files, etc. Known examples of the storage service system 70 include OneDrive (registered trademark), Google Workspace (registered trademark), and DropBox (registered trademark). The storage service system 70 may also be an on-premise NAS (Network Attached Storage), etc.

[0057] The voice recognition service system 80 provides a service of performing voice recognition on voice data and converting it into text data. The voice recognition service system 80 may be a general-purpose commercial service or may be part of the functions of the information processing system 50.

[0058] <Hardware configuration example> The hardware configuration of the information processing system 50 and the terminal device 10 according to this embodiment will be described with reference to FIG.

[0059] <<Information processing system and terminal device>> Fig. 4 is a diagram showing an example of the hardware configuration of an information processing system 50 and a terminal device 10 according to this embodiment. As shown in Fig. 4, the information processing system 50 and the terminal device 10 are constructed by a computer, and include a CPU 501, a ROM 502, a RAM 503, a HD (Hard Disk) 504, an HDD (Hard Disk Drive) controller 505, a display 506, an external device connection I / F (Interface) 508, a network I / F 509, a bus line 510, a keyboard 511, a pointing device 512, an optical drive 514, and a media I / F 516.

[0060] Of these, the CPU 501 controls the overall operation of the information processing system 50 and the terminal device 10. The ROM 502 stores programs, such as an IPL, used to drive the CPU 501. The RAM 503 is used as a work area for the CPU 501. The HD 504 stores various data, such as programs. The HDD controller 505 controls the reading and writing of various data from and to the HD 504 under the control of the CPU 501. The display 506 displays various information, such as a cursor, menus, windows, characters, or images. The external device connection I / F 508 is an interface for connecting various external devices. In this case, external devices include, for example, a USB (Universal Serial Bus) memory or a printer. The network I / F 509 is an interface for data communication using a network. The bus line 510 is an address bus, a data bus, or the like, for electrically connecting the components, such as the CPU 501, shown in FIG. 4.

[0061] The keyboard 511 is a type of input means having multiple keys used to input characters, numbers, various instructions, etc. The pointing device 512 is a type of input means for selecting and executing various instructions, selecting a processing target, moving a cursor, etc. The optical drive 514 controls reading and writing of various data from an optical storage medium 513, which is an example of a removable storage medium. The optical storage medium 513 may be a CD, a DVD, Blu-ray (registered trademark), etc. The media I / F 516 controls reading and writing (storing) of data from a storage medium 515, such as a flash memory.

[0062] <<Meeting Devices>> The hardware configuration of the meeting device 60 will be described using Figure 5. Figure 5 is an example of a hardware configuration diagram of a meeting device 60 capable of capturing 360-degree video. In the following, the meeting device 60 is described as a device that uses an imaging element to capture 360-degree video around the device at a predetermined height, but the imaging element may be one, two, or any number of elements. Furthermore, it does not necessarily have to be a dedicated device; a PC, digital camera, smartphone, etc. may have substantially the same functionality by attaching a 360-degree video imaging unit to it after the fact.

[0063] As shown in Figure 5, the meeting device 60 is composed of an imaging unit 601, an image processing unit 604, an imaging control unit 605, a microphone 608, a sound processing unit 609, a CPU (Central Processing Unit) 611, a ROM (Read Only Memory) 612, an SRAM (Static Random Access Memory) 613, a DRAM (Dynamic Random Access Memory) 614, an operation unit 615, an external device connection I / F 616, a communication unit 617, an antenna 617a, an audio sensor 618, and a concave terminal 621 for Micro USB.

[0064] Of these, the imaging unit 601 includes a wide-angle lens (so-called fisheye lens) 602 with a 360° angle of view for forming a hemispherical image, and imaging elements (image sensors) 603 provided in correspondence with each wide-angle lens. The imaging elements 603 include an image sensor such as a CMOS (Complementary Metal Oxide Semiconductor) sensor or a CCD (Charge Coupled Device) sensor that converts the optical image captured by the fisheye lens 602 into image data in the form of an electrical signal and outputs it, a timing generation circuit that generates horizontal or vertical synchronization signals and pixel clocks for the image sensor, and a group of registers in which various commands and parameters required for the operation of the imaging elements are set.

[0065] An imaging element 603 (image sensor) of the imaging unit 601 is connected to an image processing unit 604 via a parallel I / F bus. On the other hand, the imaging element 603 of the imaging unit 601 is connected to an imaging control unit 605 via a serial I / F bus (such as an I2C bus). The image processing unit 604, imaging control unit 605, and sound processing unit 609 are connected to a CPU 611 via a bus 610. Furthermore, a ROM 612, an SRAM 613, a DRAM 614, an operation unit 615, an external device connection I / F 616, a communication unit 617, an audio sensor 618, and the like are also connected to the bus 610.

[0066] The image processing unit 604 takes in the image data output from the image sensor 603 via a parallel I / F bus, performs predetermined processing on the image data, and creates data for a panoramic image and a speaker image from the fisheye video. Furthermore, the image processing unit 604 synthesizes the panoramic image and the speaker image, etc., and outputs a single video.

[0067] The imaging control unit 605 generally uses the I2C bus to set commands and the like in the registers of the imaging element 603, with the imaging control unit 605 acting as a master device and the imaging element 603 acting as a slave device. Necessary commands and the like are received from the CPU 611. The imaging control unit 605 also uses the I2C bus to retrieve status data and the like from the registers of the imaging element 603 and send it to the CPU 611.

[0068] Furthermore, the imaging control unit 605 instructs the imaging element 603 to output image data when the imaging start button on the operation unit 615 is pressed or when an imaging start instruction is received from a PC. Some meeting devices 60 have a preview display function or a function that supports video display on a display (for example, a PC or smartphone display). In this case, the image data is output from the imaging element 603 continuously at a predetermined frame rate (frames / minute).

[0069] As will be described later, the imaging control unit 605 also functions as a synchronization control means that cooperates with the CPU 611 to synchronize the output timing of image data from the imaging element 603. In this embodiment, the meeting device 60 is not provided with a display, but a display unit may be provided.

[0070] Microphone 608 converts sound into sound (signal) data. Sound processing unit 609 takes in the audio data output from microphones 608a, 608b, and 608c via the I / F bus, mixes this audio data, and performs predetermined processing. Sound processing unit 609 also determines the direction of the sound source (speaker) from the audio level (volume) input from microphones 608a, 608b, and 608c.

[0071] The CPU 611 controls the overall operation of the meeting device 60 and executes necessary processing. The ROM 612 stores various programs for operating the meeting device 60. The SRAM 613 and DRAM 614 are work memories that store programs executed by the CPU 611, data in the middle of processing, etc. In particular, the DRAM 614 stores image data in the middle of processing by the image processing unit 604 and data of processed equirectangular projection images.

[0072] The operation unit 615 is a general term for operation buttons such as an imaging start button 615a. By operating the operation unit 615, the user can start imaging or recording, as well as turn the power on / off, establish communication connections, and input settings such as various imaging modes and imaging conditions.

[0073] The external device connection I / F 616 is an interface for connecting various external devices. In this case, the external device is, for example, a PC (Personal Computer). The video data and image data stored in the DRAM 614 are transmitted to an external terminal via the external device connection I / F 616 or recorded on external media.

[0074] The communication unit 617 may communicate with a cloud server via the Internet using a wireless communication technology such as Wi-Fi via an antenna 617a provided in the meeting device 60, and transmit the stored video data and image data to the cloud server. The communication unit 617 may also be capable of communicating with nearby devices using a short-range wireless communication technology such as BLE (Bluetooth Low Energy, registered trademark) or NFC.

[0075] The audio sensor 618 is a sensor that acquires 360° audio information to identify from which direction a loud audio is being input within 360° around (horizontal plane) the meeting device 60. The audio processing unit 609 identifies the strongest direction based on the input 360° audio parameters and outputs the audio input direction within 360°.

[0076] Note that other sensors (such as a direction / acceleration sensor or GPS) may calculate direction, position, angle, acceleration, etc., and use this information for image correction or adding location information.

[0077] The image processing unit 604 also creates a panoramic image in the following way: The CPU 611 performs predetermined camera image processing such as Bayer conversion (RGB interpolation processing) on ​​the RAW data input from the image sensor that inputs the spherical image to create a fisheye image (curved image). It then performs DeWarp processing (distortion correction processing) on ​​the created fisheye image (curved image) to create a panoramic image (flat image) that captures a 360-degree view of the area around the meeting device 60.

[0078] The CPU 611 creates speaker images in the following manner. The CPU 611 creates speaker images by cutting out the speaker from a panoramic image (flat image) that captures 360° of the surrounding area. The CPU 611 cuts out the speaker image from the panoramic image, using the voice input direction identified from the 360° output using the audio sensor 618 and the sound processing unit 609 as the speaker's direction. In this case, the method for cutting out a person's image from the voice input direction is to cut out a 30° area centered on the identified voice direction from the 360°, and perform face detection within that area to cut out the image. The CPU 611 further identifies the speaker images for a specific number of people (e.g., three people) who have spoken most recently from the cut-out speaker images.

[0079] The panoramic image and the one or more speaker images may be sent separately to the information recording application 41, or the meeting device 60 may create a single image from them and send it to the information recording application 41. In this embodiment, the panoramic image and the one or more speaker images are sent separately from the meeting device 60 to the information recording application 41.

[0080] Figure 6 is a diagram illustrating the imaging range of the meeting device 60. As shown in Figure 6(a), the meeting device 60 captures an image in a 360° range in the horizontal direction. As shown in Figure 6(b), the imaging range of the meeting device 60 is 0° in the horizontal direction at the height of the meeting device 60, and a predetermined angle above and below.

[0081] FIG. 7 is a diagram illustrating the cutting out of a panoramic image and a speaker image. As shown in FIG. 7, the image captured by the meeting device 60 forms a portion 110 of a sphere and therefore has a three-dimensional shape. As shown in FIG. 6(b), the meeting device 60 performs perspective projection transformation by dividing the angle of view into predetermined angles in the vertical direction and the horizontal direction. By performing perspective projection transformation across the entire 360° horizontal direction without gaps, a predetermined number of planar images are obtained, and a panoramic image 111 is obtained by connecting the predetermined number of planar images horizontally. In addition, the meeting device 60 performs face detection within a predetermined range from the panoramic image centered on the direction of the voice, and creates a speaker image 112 by cutting out an image 15° left and right from the center of the face (30° overall).

[0082] <<Electronic Whiteboard>> Fig. 8 is a hardware configuration diagram of the electronic whiteboard 2. As shown in Fig. 8, the electronic whiteboard 2 includes a CPU (Central Processing Unit) 401, a ROM (Read Only Memory) 402, a RAM (Random Access Memory) 403, an SSD (Solid State Drive) 404, a network I / F 405, and an external device connection I / F (Interface) 406.

[0083] Of these, the CPU 401 controls the overall operation of the electronic whiteboard 2. The ROM 402 stores programs used to start up the OS, such as an IPL (Initial Program Loader). The RAM 403 is used as a work area for the CPU 401. The SSD 404 stores various data, such as programs for the electronic whiteboard 2. The network I / F 405 controls communication with a communication network. The external device connection I / F 406 is an interface for connecting various external devices. In this case, the external devices are, for example, a USB (Universal Serial Bus) memory 430 and external devices (a microphone 440, a speaker 450, and a camera 460).

[0084] The electronic whiteboard 2 also includes a capture device 411, a GPU 412, a display controller 413, a contact sensor 414, a sensor controller 415, an electronic pen controller 416, a short-range communication circuit 419, an antenna 419a of the short-range communication circuit 419, a power switch 422, and selection switches 423.

[0085] Of these, the capture device 411 displays display information on the display of an external PC (Personal Computer) 470 as still images or moving images. The GPU (Graphics Processing Unit) 412 is a semiconductor chip that specializes in graphics. The display controller 413 controls and manages the screen display to output an image output from the GPU 412 to a display 480 or the like. The contact sensor 414 detects contact of the electronic pen 490, the user's hand H, or the like on the display 480. The sensor controller 415 controls the processing of the contact sensor 414. The contact sensor 414 inputs and detects coordinates using an infrared blocking method. This coordinate input and coordinate detection method involves two light receiving and emitting devices installed at both ends of the upper side of the display 480 emitting multiple infrared rays parallel to the display 480, which are reflected by reflecting members installed around the display 480 and return along the same optical path as the light emitted by the light receiving element. The light receiving and emitting device serving as the contact sensor 414 outputs the position (position on the light receiving element) where the infrared light is blocked by an object to the sensor controller 415, and the sensor controller 415 identifies the coordinate position, which is the contact position of the object, from these two pieces of position information. The electronic pen controller 416 determines whether the pen tip or the pen tail has touched the display 480 by communicating with the electronic pen 490 via Bluetooth. The short-range communication circuit 419 is a communication circuit such as NFC (Near Field Communication) or Bluetooth (registered trademark). The power switch 422 is a switch for switching the power of the electronic whiteboard 2 on and off. The selection switches 423 are a group of switches for adjusting, for example, the brightness and color of the display 480.

[0086] Furthermore, the electronic whiteboard 2 includes a bus line 410. The bus line 410 is an address bus, a data bus, or the like for electrically connecting the components such as the CPU 401 shown in FIG.

[0087] The contact sensor 414 is not limited to an infrared blocking type, and may be a capacitive touch panel that identifies the contact position by detecting changes in capacitance. The contact sensor 414 may be a resistive film type touch panel that identifies the contact position by voltage changes across two opposing resistive films. The contact sensor 414 may be an electromagnetic induction type touch panel that identifies the contact position by detecting electromagnetic induction caused by an object touching the display unit. In addition, the contact sensor 414 may use various detection means. Furthermore, the electronic pen controller 416 may determine whether or not the part of the electronic pen 490 that the user grips or other parts of the electronic pen are touched, in addition to the pen tip and pen butt.

[0088] <About the function> Next, the functional configuration of the record information creation system 100 will be described with reference to Fig. 9. Fig. 9 is an example of a functional block diagram that explains the functions of the terminal device 10, the meeting device 60, and the information processing system 50 in the record information creation system 100 by dividing them into blocks.

[0089] <<Terminal Device>> The information recording application 41 running on the terminal device 10 includes a communication unit 11, an operation reception unit 12, a display control unit 13, an application screen acquisition unit 14, an audio acquisition unit 15, a device communication unit 16, a recording control unit 17, an audio data processing unit 18, a recording / playback unit 19, an upload unit 20, an edit processing unit 21, a code analysis unit 22, and a time measurement unit 25. Each of these units included in the terminal device 10 is a function or a means for performing a function that is realized when any of the components shown in FIG. 4 operates in response to an instruction from the CPU 501 in accordance with the information recording application 41 loaded from the HD 504 to the RAM 503. The terminal device 10 also includes a storage unit 1000 implemented by the HD 504 and the like shown in FIG. 4. The storage unit 1000 includes an information storage unit 1001.

[0090] The communication unit 11 communicates various types of information with the information processing system 50 via a network. For example, the communication unit 11 receives a list of remote conferences from the information processing system 50 and transmits a request for recognizing voice data to the information processing system 50.

[0091] The display control unit 13 displays various screens serving as a user interface in the information recording application 41 in accordance with screen transitions set in the information recording application 41. The operation accepting unit 12 accepts various operations for the information recording application 41.

[0092] The application screen acquisition unit 14 acquires the desktop screen or the screen of the application selected by the user from the OS (Operating System), etc. If the application selected by the user is the remote conference application 42, the screen generated by the remote conference application 42 (images of each location, images of documents, etc.) is obtained.

[0093] The audio acquisition unit 15 acquires audio data received by the terminal device 10 from the teleconference application 42 during the teleconference. Note that the audio acquired by the audio acquisition unit 15 does not include audio collected by the terminal device 10, but only audio data received during the teleconference. This is because the meeting device 60 separately collects audio.

[0094] The device communication unit 16 communicates with the meeting device 60 using a USB cable or the like. The device communication unit 16 may communicate with the meeting device 60 via wireless LAN, Bluetooth (registered trademark), or the like. The device communication unit 16 receives the panoramic image and the speaker image from the meeting device 60, and transmits the audio data acquired by the audio acquisition unit 15 to the meeting device 60. The device communication unit 16 receives the audio data synthesized by the meeting device 60.

[0095] The recording control unit 17 creates a combined image by combining the panoramic image and speaker image received by the device communication unit 16 and the application screen acquired by the application screen acquisition unit 14. The recording control unit 17 also creates a combined image video by connecting the repeatedly created combined images in chronological order, and creates a combined image video with audio by combining the audio data synthesized by the meeting device 60 with the combined image video.

[0096] The audio data processing unit 18 requests the information processing system 50 to convert the audio data combined with the combined image video extracted by the recording control unit 17, or the synthesized audio data received from the meeting device 60, into text data.

[0097] The recording / playback unit 19 plays back the combined image moving image. The combined image moving image is stored in the terminal device 10 while being recorded, and is then uploaded to the information processing system 50.

[0098] When the remote conference ends, the upload unit 20 transmits the combined image moving image to the information processing system 50.

[0099] The editing processing unit 21 executes editing of the combined image moving image (deleting a part, splicing, etc.) in response to a user operation.

[0100] The code analysis unit 22 detects the two-dimensional code included in the panoramic image, and analyzes the two-dimensional code to obtain device identification information.

[0101] The time measurement unit 25 measures the time from when the information recording application 41 is started until when the two-dimensional code is received from the meeting device 60. When a certain time has elapsed, the time measurement unit 25 notifies the display control unit 13 of this, and the display control unit 13 displays an error dialog.

[0102] Fig. 10 shows video recording information stored in the information storage unit 1001. The video recording information includes items such as a conference ID, a recording ID, an update date and time, a title, an upload, and a save destination. When a user logs in to the information processing system 50, the information recording application 41 downloads the conference information from the conference information storage unit 5001 of the information processing system 50. The conference ID and other information included in the conference information are reflected in the video recording information. The video recording information in Fig. 10 is held by a terminal device 10 operated by a certain user.

[0103] The conference ID is identification information for identifying the remote conference that has been held. The conference ID is assigned when the remote conference is scheduled and registered in the conference management system 9, or is assigned by the information processing system 50 in response to a request from the information recording application 41.

[0104] The recording ID is identification information for identifying the combined image video recorded during a remote conference. The recording ID is assigned by the meeting device 60, but may also be assigned by the information recording application 41 or the information processing system 50. A different recording ID is assigned to the same conference ID when recording stopped during the remote conference but was resumed for some reason.

[0105] The update date and time is the date and time when the combined image video was updated (recording ended). If the combined image video was edited, this is the edit date and time.

[0106] The title is the name of the conference. It may be set when the conference is registered in the conference management system 9, or may be set arbitrarily by the user.

[0107] The "upload" indicates whether the combined image video has been uploaded to the information processing system 50.

[0108] The destination indicates the location (URL or file path) where the combined image video and text data are saved in the storage service system 70. Therefore, the user can view the uploaded combined image video at will. The combined image video and text data are saved with separate file names, for example, following the URL.

[0109] <<Meeting Devices>> Returning to Fig. 9, the meeting device 60 has a terminal communication unit 61, a panoramic image creation unit 62, a speaker image creation unit 63, a sound collection unit 64, and a voice synthesis unit 65. Each of these units of the meeting device 60 is a function or a means for performing the function, which is realized when any of the components shown in Fig. 5 operates in response to an instruction from the CPU 611 in accordance with a program loaded from the ROM 612 to the DRAM 614.

[0110] The terminal communication unit 61 communicates with the terminal device 10 using a USB cable or the like. The terminal communication unit 61 may be connected not only by a wired cable but also by wireless LAN, Bluetooth (registered trademark), or the like as long as it can communicate with the terminal device 10.

[0111] The panoramic image creation unit 62 creates a panoramic image. The speaker image creation unit 63 creates a speaker image. These creation methods have been explained with reference to Figures 6 and 7. The panoramic image creation unit 62 also serves as an acquisition unit that acquires image data.

[0112] The sound collection unit 64 converts the sound picked up by the microphone of the meeting device 60 into digital sound data. As a result, the sound of what the users and participants say is collected at the base on the terminal device 10 side.

[0113] The voice synthesis unit 65 synthesizes the voice transmitted from the terminal device 10 with the voice collected by the voice collection unit 64. As a result, the voices spoken at the other locations 101 and the voices spoken at the own location 102 are combined.

[0114] <<Information Processing System>> The information processing system 50 has a communication unit 51, an authentication unit 52, a screen generation unit 53, a communication management unit 54, a device management unit 55, and a text conversion unit 56. Each of these units of the information processing system 50 is a function or a means for performing a function that is realized when any of the components shown in Fig. 4 operates in response to an instruction from the CPU 501 in accordance with a program loaded from the HD 504 to the RAM 503. The information processing system 50 also has a storage unit 5000 constructed by the HD 504 etc. shown in Fig. 4. The storage unit 5000 has a conference information storage unit 5001, a recording information storage unit 5002, and an association information storage unit 5003 constructed therein.

[0115] The communication unit 51 transmits and receives various types of information to and from the terminal device 10. For example, the communication unit 51 transmits a list of remote conferences to the terminal device 10 and receives a request to recognize voice data from the terminal device 10.

[0116] The authentication unit 52 authenticates the user operating the terminal device 10. For example, the authentication unit 52 authenticates the user based on whether or not authentication information (user ID and password) included in an authentication request received by the communication unit 51 matches authentication information stored in advance. Note that the authentication information may be an IC card number, biometric authentication information such as face or fingerprint, or the like. The authentication unit 52 may also perform authentication using an external authentication system or an authentication method such as OAUTH.

[0117] The screen generation unit 53 generates screen information to be displayed by a Web application on the terminal device 10. The screen information is created using HTML, XML, CSS (Cascade Style Sheet), JavaScript (registered trademark), and the like.

[0118] The communication management unit 54 acquires information about the remote conference from the conference management system 9 using each user's account or a system account assigned to the information processing system 50. The communication management unit 54 stores the conference information of the reserved conference in the conference information storage unit in association with the conference ID. The communication management unit 54 also acquires conference information that users belonging to the tenant have permission to view. Since a conference ID is set for each conference, the remote conference and recorded information are associated with each other by the conference ID.

[0119] When the device management unit 55 receives the device identification information of the electronic whiteboard 2 and the meeting device 60 used in a conference, it associates them and stores them in the association information storage unit 5003. Therefore, the conference ID, the device identification information of the electronic whiteboard 2, and the device identification information of the meeting device 60 are associated with each other. Since the conference ID is also associated with the combined image video, the handwritten data input on the electronic whiteboard 2 and the combined image video are also associated with each other. Furthermore, when the recording ends (when the conference ends), the device management unit 55 deletes the association from the association information storage unit 5003.

[0120] The text conversion unit 56 converts the voice data requested by the terminal device 10 to be converted into text data into text data using an external voice recognition service system 80. The text conversion unit 56 may also perform the conversion itself.

[0121] 11 shows an example of conference information stored in the conference information storage unit 5001 managed by the communication management unit 54. The communication management unit 54 can use the above account to obtain a list of remote conferences that the user belonging to the tenant has viewing authority for. In this embodiment, a remote conference is used as an example, but the list of remote conferences also includes conferences that are held in only one conference room.

[0122] Conference information is managed by a conference ID and is associated with participants, title (conference name), start date and time, end date and time, location, etc. These are examples of conference information, and conference information may include other information as well.

[0123] Participant items are the participants in the meeting.

[0124] The title item indicates the content of the meeting, such as the meeting name and agenda.

[0125] The start date and time item is the date and time when the conference is scheduled to start.

[0126] The end date and time item is the date and time when the conference is scheduled to end.

[0127] The location item is the location where the meeting is held, such as a conference room, a branch office name, or a building.

[0128] The item of electronic whiteboard information is the identification information of the electronic whiteboard 2 used in the meeting.

[0129] The meeting device item is identification information of the meeting device 60 used in the conference.

[0130] As shown in FIGS. 10 and 11, the combined image video recorded at the conference is identified by the conference ID.

[0131] The recording information stored in the recording information storage unit 5002 may be the same as that shown in Fig. 10. However, the information processing system 50 has a list of combined image videos recorded by all users belonging to the tenant.

[0132] 12 shows the association information stored in the association information storage unit 5003, in which the device identification information of the electronic whiteboard 2 and the device identification information of the meeting device 60 are associated with the conference ID. This association information is held from the time the information recording application 41 transmits the device identification information to the information processing system 50 until the recording is completed.

[0133] <<Electronic Whiteboard>> 13 is a functional block diagram explaining the functions of the electronic whiteboard 2 by dividing them into blocks. The electronic whiteboard 2 has a contact position detection unit 31, a drawing data generation unit 32, a data recording unit 33, a display control unit 34, a code generation unit 35, and a communication unit 36. Each function of the electronic whiteboard 2 is a function or means realized when any of the components shown in FIG. 8 operates in response to an instruction from the CPU 401 in accordance with a program loaded from the SSD 404 onto the RAM 403.

[0134] The contact position detection unit 31 detects the coordinates of the position where the electronic pen 490 contacts the contact sensor 414. The drawing data generation unit 32 acquires the coordinates where the pen tip of the electronic pen 490 contacts from the contact position detection unit 31. The drawing data generation unit 32 generates stroke data by connecting this sequence of coordinate points by interpolating them.

[0135] The display control unit 34 displays handwritten data, menus for user operation, and the like on the display.

[0136] The data recording unit 33 stores handwritten data written on the electronic whiteboard 2, shapes such as circles and triangles, stamps such as "Done," PC screens, files, etc. in the object information storage unit 3002. Handwritten data, shapes, images such as PC screens, files, etc. are handled as objects. For handwritten data, a group of stroke data becomes one object depending on whether the handwriting has been interrupted in time or whether the handwriting location has changed in distance.

[0137] The communication unit 36 ​​is connected to Wi-Fi or a LAN and communicates with the information processing system 50. The communication unit 36 ​​transmits object information to the information processing system 50, receives object information stored in the information processing system 50 from the information processing system 50, and displays the object information on the display 480.

[0138] The code generation unit 35 generates a two-dimensional code by encoding the device identification information of the electronic whiteboard 2 stored in the device information storage unit 3001 and information indicating that the device is usable for a conference into a two-dimensional pattern. The code generation unit 35 can also encode the device identification information of the electronic whiteboard 2 and information indicating that the device is usable for a conference into a barcode. The device identification information may be a serial number or a UUID (Universally Unique Identifier), or may be set by the user. The code generation unit 35 also serves as an output unit that outputs a two-dimensional code or a barcode.

[0139] The electronic whiteboard 2 also has a storage unit 3000 constructed using the SSD 404 and the like shown in Fig. 8. The storage unit 3000 includes a device information storage unit 3001 and an object information storage unit 3002.

[0140] FIG. 14 shows the device identification information and the like stored in the device information storage unit 3001. The device identification information is the identification information of the electronic whiteboard 2. The IP address is an IP address that allows other devices to connect to the electronic whiteboard 2 via a network. The password is used to authenticate other devices when they connect to the electronic whiteboard 2.

[0141] 15 is a diagram illustrating object information stored in the object information storage unit 3002. The object information is information for managing objects displayed on the electronic whiteboard 2. The object information is transmitted to the information processing system 50 and used as minutes of the meeting. If an electronic whiteboard 2 is located at another location when a remote conference is held, the object information is shared.

[0142] The conference ID is identification information of the conference notified by the information processing system 50.

[0143] · Object ID is identification information that identifies an object.

[0144] Type is the type of object, and can be handwritten, text, shape, image, etc. Handwritten text is stroke data (a sequence of coordinate points). Text is a string of characters (character code) entered from a soft keyboard. Text is sometimes called text data. Shapes are geometric shapes such as triangles and squares. Images are image data such as JPEG, PNG, TIFF, etc. imported from a PC or the Internet.

[0145] One screen of the electronic whiteboard 2 is called a page. The item on a page is its page number.

[0146] The coordinates indicate the position of the object relative to a predetermined origin on the electronic whiteboard 2. The object position is, for example, the upper left vertex of the bounding rectangle of the object. The coordinates are expressed, for example, in pixel units of the display.

[0147] The size is the width and height of the object's bounding rectangle.

[0148] <Screen transition> Next, several screens that the terminal device 10 displays during a remote conference will be described with reference to Fig. 16 to Fig. 19. Fig. 16 shows an initial screen 200 after login that is displayed by the information recording application 41 running on the terminal device 10. The user of the terminal device 10 connects the information recording application 41 to the information processing system 50. When the user inputs authentication information and successfully logs in, the initial screen 200 of Fig. 16 is displayed.

[0149] The initial screen 200 has a fixed display button 201, a front change button 202, a panoramic image 203, one or more speaker images 204a to 204c (hereinafter, referred to as speaker images 204 when no distinction is made), and a recording start button 205. If the meeting device 60 is already running and capturing images of the surroundings at the time of login, the panoramic image 203 and speaker image 204 created by the meeting device 60 are displayed on the initial screen 200. Therefore, the user can decide whether to start recording while viewing these. If the meeting device 60 is not running (not capturing images), the panoramic image 203 and speaker image 204 are not displayed.

[0150] The information recording application 41 may display speaker images 204 of all participants based on all faces detected from the panoramic image 203, or may display only the speaker images 204 of the N most recently spoken participants. Fig. 16 shows an example in which speaker images 204 of up to three people are displayed. Until a participant speaks, there may be no speaker image 204 (the speaker images are added one by one as the participant speaks), or three speaker images 204 of participants in a predetermined direction may be displayed (the speaker images are replaced as the participant speaks).

[0151] When no one is speaking, such as immediately after the meeting device 60 is started up, a predetermined direction (0°, 120°, 240°, etc.) within the horizontal 360° is created as the speaker image 204. If a fixed display, which will be described later, is set, the fixed display setting takes priority.

[0152] The fixed display button 201 is a button that allows the user to perform an operation to fix a certain area of ​​the panoramic image 203 in close-up as a speaker image 204 .

[0153] The front change button 202 is a button that allows the user to perform an operation to change the front of the panoramic image 203 (because a panoramic image covers 360° horizontally, the right and left edges are aligned). The user can slide the panoramic image 203 left and right with a pointing device to determine which participant will be shown in the front. The user's operation is transmitted to the meeting device 60, which then changes the angle of the front within the 360° horizontal direction, creates a panoramic image, and transmits it to the terminal device 10.

[0154] When the user presses the record start button 205, the information recording application 41 displays the recording setting screen 210 shown in FIG.

[0155] 17 is an example of a recording setting screen 210 displayed by the information recording application 41. On the recording setting screen 210, the user can set whether to record (whether to include in the recording) the panoramic image and speaker image created by the meeting device 60, and the desktop screen or the screen of a running application of the terminal device 10. If the information recording application 41 does not record either the panoramic image and speaker image, or the desktop screen or the screen of a running application, only the audio (audio output by the terminal device 10 + audio collected by the meeting device 60) is recorded.

[0156] The camera toggle button 211 is a button for switching on and off the recording of the panoramic image and the speaker image created by the meeting device 60. The camera toggle button 211 may be set to record the panoramic image and the speaker image separately.

[0157] The PC screen toggle button 212 is a button that switches on and off recording of the desktop screen of the terminal device 10 and the screen of an application running on the terminal device 10. When the PC screen toggle button 212 is in the on state, the desktop screen is recorded.

[0158] If the user wishes to record the screen of an application, the user further selects the application in the application selection field 213. The application selection field 213 displays the names of applications currently running on the terminal device 10 in a pull-down menu. The user can select the application to record. The information recording application 41 obtains this application name from the OS. The information recording application 41 can display only those applications that have a UI (screen) among the currently running applications. The selected applications may include the remote conference application 42. Therefore, the information recording application 41 can record videos of materials displayed in the remote conference application 42 and participants at each location. In addition, various applications are displayed in the pull-down menu, such as a presentation application, a word processing application, a spreadsheet application, and a web browser application. Therefore, the user can flexibly select the application screens to be included in the combined image video.

[0159] When recording on an app basis, the user can select multiple apps, and the information recording app 41 can record the screens of all the selected apps.

[0160] When both the camera toggle button 211 and the PC screen toggle button 212 are off, the recording content confirmation window 214 displays "Only audio will be recorded." This audio includes the audio output by the terminal device 10 (audio received by the remote conference app 42 from other locations 101) and the audio collected by the meeting device 60. In other words, as long as a remote conference is being held, the audio from the remote conference app 42 and the audio from the meeting device 60 are saved regardless of whether images are recorded. However, the user may be able to selectively stop saving the audio from the remote conference app 42 and the audio from the meeting device 60 using user settings.

[0161] The combined image moving image is recorded as follows according to the combination of on / off of the camera toggle button 211 and the PC screen toggle button 212. The combined image moving image is also displayed in real time in the recording content confirmation window 214.

[0162] When the camera toggle button 211 is on and the PC screen toggle button 212 is off, the panoramic image captured by the meeting device 60 and the speaker image are displayed in the recording content confirmation window 214 .

[0163] When the camera toggle button 211 is off and the PC screen toggle button 212 is on (the screen has also been selected), the recording content confirmation window 214 displays the desktop screen or the screen of the selected application.

[0164] When the camera toggle button 211 is on and the PC screen toggle button 212 is on, the panoramic image and speaker image captured by the meeting device 60, as well as the desktop screen and the screen of the selected application, are displayed side by side in the recording content confirmation window 214.

[0165] Therefore, there are cases where the panoramic image, speaker image, and app screen are not recorded, or where the panoramic image, speaker image, and app screen are not recorded at all, but in this embodiment, for convenience, the image created by the information recording application 41 is referred to as a combined image video.

[0166] Furthermore, the recording setting screen 210 has a check box 215 along with the message "Automatically transcribe recording after uploading." The recording setting screen 210 also has a Start Recording Now button 216. When the user checks the check box 215, text data obtained by converting comments made during the remote conference is attached to the recorded video. In this case, after the recording is completed, the information recording application 41 uploads the audio to the information processing system 50 along with a request to convert it into text data. Furthermore, when the user presses the Start Recording Now button 216, the recording in progress screen 220 shown in FIG. 18 is displayed.

[0167] FIG. 18 is an example of a recording screen 220 that the information recording application 41 displays during recording. Note that the explanation of FIG. 18 will mainly focus on the differences from FIG. 16. The recording screen 220 displays a combined image video being recorded in real time under the conditions set by the user on the recording setting screen 210. The recording screen 220 of FIG. 18 is displayed when the camera toggle button 211 is on and the PC screen toggle button 212 is off, and displays a panoramic image 203 and a speaker image 204 (both of which are videos) created by the meeting device 60. The recording screen 220 displays a recording icon 225, a pause button 226, and an end recording button 227.

[0168] The pause button 226 is a button for stopping recording, and after stopping, recording can also be resumed. The end recording button 227 is a button for ending recording. The pause button 226 does not change the recording ID, but the end recording button 227 changes the recording ID. After pausing, when recording is resumed, the user can reset the recording conditions set on the recording setting screen 210. In this case, the information recording application 41 may create multiple recording files each time recording is stopped (for example, by pressing the end recording button 227), or may combine multiple files so that they form a single continuous video (for example, by pressing the pause button 226). Furthermore, when the information recording application 41 plays back a combined image video, it may play back multiple recording files continuously as a single video.

[0169] Furthermore, the recording in progress screen 220 has an Get Information From Calendar button 221, a conference name field 222, a time field 223, and a location field 224. The Get Information From Calendar button 221 is a button that the user uses to obtain conference information from the conference management system 9. When the Get Information From Calendar button 221 is pressed, the information recording application 41 obtains and displays a list of conferences that the user has permission to view from the information processing system 50. The user selects the remote conference to be held from the conference list. This causes the conference information to be reflected in the conference name field 222, time field 223, and location field 224. The conference name field 222 reflects the title of the conference information, the time field 223 reflects the start time and end time, and the location field 224 reflects the location. Furthermore, the conference information and recording information in the conference management system are associated with each other by a conference ID.

[0170] Once the remote conference is over and the user stops recording, a combined image video with audio is created.

[0171] FIG. 19 is an example of the conference list screen 230 displayed by the information recording application 41. The conference list screen 230 is a list of conferences, and can display a list of recorded information recorded during remote conferences. It also includes conferences that were held only in a certain conference room, regardless of whether they were remote conferences. The conference list screen 230 displays conference information that the logged-in user has permission to view in the conference information storage unit 5001. Information on video recording information stored in the information storage unit 1001 may be further integrated.

[0172] The conference list screen 230 is displayed when the user selects the conference list tab 231 on the initial screen 200 in FIG. 16. The conference list screen 230 displays a list 236 of recorded information that the user has permission to view. The conference creator (minutes creator) can set viewing permissions for participants. The conference list may be a list of stored recorded information or a list of conference schedules and conference data.

[0173] The conference list screen 230 has the following items: a check box 232 , an update date and time 233 , a title 234 , and a status 235 .

[0174] The checkboxes 232 are used to select recorded files. The checkboxes 232 are used when the user wants to delete recorded files all at once.

[0175] The update date and time 233 indicates the start time of recording the combined image video. If the video has been edited, the edit date and time may be used.

[0176] The title 234 is the title of the meeting (agenda, etc.). This may be transcribed from the meeting information or set by the user.

[0177] The status 235 indicates whether the combined image video has been uploaded to the information processing system 50. If it has not been uploaded, "Local PC" is displayed, and if it has been uploaded, "Uploaded" is displayed. If it has not been uploaded, an upload button is displayed. If there is a combined image video that has not been uploaded, it is recommended that the information recording application 41 automatically uploads it when the user logs in to the information processing system 50.

[0178] When the user selects an arbitrary title or the like from the list 236 of combined image videos with a pointing device, the information recording application 41 displays a recording playback screen, but this is not shown in this embodiment. On the recording playback screen, it is possible to play back the combined image video, etc.

[0179] It is desirable that users be able to narrow down conferences by update date and time, title, keywords, etc. Also, if there are a large number of conferences displayed and it is difficult to find the relevant conference, it is desirable that a search function be provided that allows users to input words and phrases to narrow down recorded information based on words contained in the remarks made at the conference or the title, etc. The search function allows users to find the desired recorded information in a short time even when there is a large amount of recorded information. Furthermore, on the conference list screen, users may be able to sort conferences by update date and time or title.

[0180] <Action or Processing> 20 is a sequence diagram illustrating the process in which the record information creation system 100 associates the meeting device 60 with the electronic whiteboard 2. In the explanation of FIG. 20, a user who is participating in a meeting is trying to participate in the meeting by using the meeting device 60 and the electronic whiteboard 2 in the same meeting.

[0181] S1: When the electronic whiteboard 2 placed in the conference room used for the conference is turned on, it communicates with the information processing system 50 that has been set up in advance, specifies device identification information, and registers that it can be associated with the conference.

[0182] S2: Furthermore, the code generation unit 35 of the electronic whiteboard 2 placed in the conference room used for the meeting generates a two-dimensional code that encodes the device identification information of the electronic whiteboard 2 itself and the fact that it is the device identification information, and the display control unit 34 displays the two-dimensional code. The two-dimensional code may further include a password that the electronic whiteboard 2 uses to authenticate other devices.

[0183] S3: The user carries the terminal device 10 and the meeting device 60 and enters the conference room where the electronic whiteboard 2 is installed, and connects the terminal device 10 and the meeting device 60 with a USB cable. The meeting device 60 is started up by power supply via the USB cable or by turning on the power. This puts the meeting device 60 into standby mode. The user also starts the information recording application 41 of the terminal device 10. The information recording application 41 starts communicating with the meeting device 60, and the meeting device 60 starts capturing images and collecting sound. The panoramic image creation unit 62 of the meeting device 60 creates a panoramic image (image data) of the surrounding area including the two-dimensional code.

[0184] S4: The terminal communication unit 61 of the meeting device 60 transmits the panoramic image to the terminal device 10.

[0185] S5: The device communication unit 16 of the terminal device 10 receives the panoramic image, and the code analysis unit 22 detects the two-dimensional code displayed on the interactive whiteboard 2 from the panoramic image. The code analysis unit 22 decodes the two-dimensional code, and if it determines that the code analysis unit 22 has embedded therein information indicating that the interactive whiteboard 2 is a device that can be used in a meeting, it acquires device identification information for the interactive whiteboard 2 from the two-dimensional code. Note that the analysis of the two-dimensional code may be performed by the meeting device 60.

[0186] S6: The communication unit 11 of the information recording application 41 specifies the device identification information of the electronic whiteboard 2 and transmits a request to register for the meeting to the information processing system 50. It is preferable that the communication unit 11 further transmits the identification information of the meeting device 60 to the information processing system 50.

[0187] S7: When the communication unit 51 of the information processing system 50 receives a request to register for a conference (device identification information), the communication management unit 54 issues a conference ID. If the information recording application 41 has accepted the selection of a conference from the conference list screen 230 or the like, the communication unit 51 attaches the conference ID to the device identification information, and therefore the communication management unit 54 does not issue a conference ID.

[0188] S8: Then, the device management unit 55 associates the device identification information with the conference ID, preferably the device identification information of the meeting device 60 with the conference ID, and stores the association information in the association information storage unit 5003.

[0189] S9, S10: The communication unit 51 of the information processing system 50 notifies the terminal device 10 and the electronic whiteboard 2 of the conference ID. The communication unit 11 of the terminal device 10 receives and stores the conference ID. Similarly, upon receiving the conference ID, the communication unit 36 ​​of the electronic whiteboard 2 stores it. Note that the terminal device 10 receives at least one of the conference ID or device identification information in response to the request to register for the conference. The electronic whiteboard 2 and the information processing system 50 may communicate using a bidirectional communication method such as WebSocket, which enables push-type communication from the information processing system 50 to the electronic whiteboard 2.

[0190] Because the electronic whiteboard 2 and the terminal device 10 have the same conference ID, the electronic whiteboard 2 and the meeting device 60 are associated with the conference. Thereafter, the terminal device 10 attaches at least one of the conference ID or the identification information of the meeting device 60 to the data it transmits, and the electronic whiteboard 2 attaches at least one of the conference ID or the device identification information to the data it transmits. In this embodiment, the conference ID is attached to the communication, but even if the device identification information or the identification information of the meeting device 60 is attached to the communication, the information processing system 50 can identify the conference ID from the association information.

[0191] 20 is used to associate recorded information with handwritten data, etc., but the electronic whiteboard 2 also has a function for sharing objects with other electronic whiteboards 2 in remote locations. The electronic whiteboard 2 and other electronic whiteboards 2 are registered on a remote conference server, allowing handwritten data, etc. to be shared in real time.

[0192] 21 is a flowchart illustrating the operation of the information processing system 50. The communication unit 51 of the information processing system 50 waits until it receives device identification information from the terminal device 10 (S101).

[0193] When the communication unit 51 receives the device identification information, the communication management unit 54 issues a conference ID (S102). When the communication unit 51 receives the conference ID attached to the device identification information, the communication management unit 54 does not issue a conference ID.

[0194] The device management unit 55 associates the conference ID with the received device identification information and stores the association information in the association information storage unit 5003 (S103). The device management unit 55 holds the association between the conference ID and the device identification information until the end of the conference (end of recording).

[0195] The communication unit 51 of the information processing system 50 transmits the conference ID to the terminal device 10 and the interactive whiteboard 2 (S104).

[0196] 22 is a flowchart illustrating the operation of the information recording application 41 that operates on the terminal device 10. This process is performed because if the information recording application 41 cannot detect the two-dimensional code because there is a person in front of the electronic whiteboard 2, the electronic whiteboard 2 cannot be associated with the meeting.

[0197] First, the information recording application 41 is started (S111).

[0198] Next, the time measurement unit 25 starts measuring the time from when the information recording application 41 is started until when the two-dimensional code is detected (S112).

[0199] The device communication unit 16 receives the panoramic image, and the code analysis unit 22 detects the two-dimensional code. The time measurement unit 25 determines whether the two-dimensional code is detected within a certain time period after startup (S113).

[0200] If a two-dimensional code is detected within a certain time period after startup (Yes in S113), the time measurement unit 25 stops measuring the time (S114).

[0201] If the two-dimensional code is not detected within a certain time after startup (No in S113), the display control unit 13 displays an error dialog in response to a notification from the time measurement unit 25 (S115).

[0202] 23 is an example of a message 301 displayed by the information recording application 41 of the terminal device 10 in an error dialog. This error dialog displays a message 301 such as, for example, "Two-dimensional code not detected. Please make sure that the two-dimensional code is displayed and that no one is in front of the electronic whiteboard 2." The user can understand from the message 301 that the two-dimensional code cannot be detected, and can take action such as moving people away from in front of the electronic whiteboard 2. An OK button 302 is a button that causes the information recording application 41 to measure time again.

[0203] Returning to FIG. 22, when the OK button is pressed in step S115 and the error dialog is closed (Yes in S116), the process returns to step S112, and the time measurement unit 25 starts measuring time.

[0204] When the user closes the error dialog, a cancel button may be provided in the error dialog, allowing the user to start the conference without associating the electronic whiteboard 2 with the conference.

[0205] <Two-dimensional codes and barcodes displayed on electronic whiteboards> 24 to 31, the two-dimensional code and barcode displayed on the interactive whiteboard 2 will be explained in more detail. If an obstacle 69 is present between the meeting device 60 and the interactive whiteboard 2, the terminal device 10 may not be able to acquire the device identification information. For this reason, it is effective to adjust the position where the interactive whiteboard 2 displays the two-dimensional code 8, as follows.

[0206] FIG. 24 shows an example of the position of the two-dimensional code 8 displayed by the interactive whiteboard 2. As described above, the two-dimensional code 8 includes device identification information and a sign indicating that it is device identification information. On the desk on which the meeting device 60 is placed, there is an obstacle 69 between the meeting device 60 and the interactive whiteboard 2. Note that the meeting device 60 does not have to be a hemispherical camera to detect the two-dimensional code 8. The interactive whiteboard 2 displays the device identification information above the center line 320 of the display screen of the interactive whiteboard 2 in the up-down direction. This makes it easier for the meeting device 60 to capture the device identification information even if there is an obstacle 69 between the meeting device 60 and the interactive whiteboard 2.

[0207] As shown in FIG. 25, the interactive whiteboard 2 may display the center of the two-dimensional code 8 above the center line 320 of the display screen of the interactive whiteboard 2 in the up-down direction.

[0208] 26, the interactive whiteboard 2 may move the two-dimensional code 8 over time. In FIG. 26, the two-dimensional code 8 moves from left to right. The interactive whiteboard 2 may move the two-dimensional code 8 while it is still displayed, or may display, erase, and then redisplay the code in a different location. This makes it easier for the meeting device 60 to capture the device identification information even if the position of the obstacle 69 is uncertain. Also, even if the interactive whiteboard 2 displays the two-dimensional code 8 in a small size to reduce the sense of oppression felt by the user, the meeting device 60 can easily capture the code.

[0209] Furthermore, the interactive whiteboard 2 may change the size of the two-dimensional code 8 while moving the two-dimensional code 8.

[0210] 27, the interactive whiteboard 2 may simultaneously display multiple two-dimensional codes 8. This makes it easier for the meeting device 60 to capture images of the remaining two-dimensional codes 8 even if some of the two-dimensional codes 8 are hidden by an obstacle 69. Note that the multiple two-dimensional codes 8 may all store the same information, or may store different information.

[0211] 28, the electronic whiteboard 2 may display a two-dimensional code 8 next to (close to) the menu 71. The menu 71 is arranged vertically across the right edge, and is displayed above the center line of the display screen of the electronic whiteboard 2 in the vertical direction, similar to FIG.

[0212] The two-dimensional code 8 is displayed near the menu 71, so that the user is less likely to feel uncomfortable. The user can use a wider area of ​​the screen.

[0213] 29, the electronic whiteboard 2 may display a two-dimensional code 8 within a menu 71. Compared to FIG. 28, this makes it easier to eliminate the sense of oppression and allows the user to use a wider screen.

[0214] Fig. 30 is a diagram illustrating a case where the camera of the meeting device 60 is a hemispherical camera. By capturing images of the configurations of Figs. 24 to 29 with a hemispherical camera with a wide horizontal field of view, it becomes easier to find the two-dimensional code 8.

[0215] It should be noted that a barcode instead of the two-dimensional code 8 can also be displayed in the same manner as in Figures 24 to 30. Figure 31 shows an example of a barcode 7.

[0216] Note that barcodes 7 are less robust against tilt and other factors than two-dimensional codes 8. For this reason, the code analysis unit 22 cuts out the black and white pattern and corrects the skew angle and pitch angle. The code analysis unit 22 also emphasizes the edges of the black bars. The code analysis unit 22 detects the barcode 7 displayed on the interactive whiteboard 2 by pattern matching the cut-out image with the pattern from the start character to the stop character at the right end, which is registered as the pattern of the barcode 7.

[0217] 24 to 31 illustrate an example in which a two-dimensional code 8 or a barcode 7 is displayed, but the electronic whiteboard 2 may display device identification information (alphabet or numbers), and the information recording application 41 may detect the device identification information by OCR processing.

[0218] <<Save combined image video>> Next, the process of saving the combined image video will be described with reference to Fig. 32. Fig. 32 is an example of a sequence diagram showing the procedure by which the information recording application 41 records the panoramic image, the speaker image, and the application screen. Since the conference ID has been transmitted to the terminal device 10 and the interactive whiteboard 2 according to Fig. 20, the information recording application 41 can record the recording information.

[0219] S21: The user operates the teleconferencing app 42 to start a teleconferencing. Here, it is assumed that the teleconferencing app 42 of the user's own base 102 and the other base 101 have started a teleconferencing. The teleconferencing app 42 of the user's own base 102 transmits images captured by a camera and audio collected by a microphone of the meeting device 60 to the teleconferencing app 42 of the other base 101. The teleconferencing app 42 of the other base 101 displays the received images on a display and outputs the received audio from a speaker. Similarly, the teleconferencing app 42 of the other base 101 transmits images captured by a camera and audio collected by a microphone of the meeting device 60 to the teleconferencing app 42 of the user's own base 102. The teleconferencing app 42 of the user's own base 102 displays the received images on a display and outputs the received audio from a speaker. Each teleconferencing app 42 repeats this process to realize the teleconferencing.

[0220] S22: The user makes settings related to recording on the recording setting screen 210 of the information recording application 41 shown in Fig. 17. The operation accepting unit 12 of the information recording application 41 accepts the settings. Here, it is assumed that the camera toggle button 211 and the PC screen toggle button 212 are both on.

[0221] If the user has already reserved a remote conference, the user can press the information acquisition button 221 on the calendar in FIG. 18 to display a list of remote conferences and select a remote conference to which the recorded video will be associated. Since the user has already logged in to the information processing system 50, the information processing system 50 identifies remote conferences that the logged-in user has permission to view. The information processing system 50 transmits a list of the identified remote conferences to the terminal device 10, and the user selects a remote conference that is currently being held or will be held in the future. This determines information about the remote conference, such as the conference ID. In this embodiment, the conference ID has already been created in the processing of FIG. 20.

[0222] S23: The user instructs the information recording application 41 to start recording (start recording now button 216). The operation reception unit 12 of the information recording application 41 receives the instruction. The display control unit 13 displays the recording screen 220.

[0223] S24: Since the conference ID has been determined, the communication unit 11 of the information recording application 41 requests the information processing system 50 for storage destination information, specifying the conference ID.

[0224] S25: The communication unit 51 of the information processing system 50 receives the request, and the communication management unit 54 transmits the destination where the combined image moving image is to be saved (the URL of the storage service system 70) to the information recording application 41 via the communication unit 51.

[0225] S26: When the communication unit 11 of the information recording application 41 receives the conference ID and the destination to save the recorded file, the recording control unit 17 determines that preparations for recording are complete and starts recording.

[0226] S27: The application screen acquisition unit 14 of the information recording application 41 requests the application screen selected by the user (more specifically, the application screen acquisition unit 14 acquires the application screen via the OS). In FIG. 32, the application selected by the user is the remote conference application 42.

[0227] S28: The recording control unit 17 of the information recording application 41 notifies the meeting device 60 of the start of recording via the device communication unit 16. When notifying, the recording control unit 17 may also notify that the camera toggle button 211 is on (requesting a panoramic image and a speaker image). Regardless of whether a request is made, the meeting device 60 sends the panoramic image and the speaker image to the information recording application 41.

[0228] S29: When the terminal communication unit 61 of the meeting device 60 receives the start of recording, it assigns a unique recording ID and returns the recording ID to the information recording application 41. The recording ID may be assigned by the information recording application 41 or may be obtained from the information processing system 50.

[0229] S30: The voice acquisition unit 15 of the information recording application 41 acquires the voice data output by the terminal device 10 (voice data received by the remote conference application 42).

[0230] S31: The device communication unit 16 transmits the voice data acquired by the voice acquisition unit 15 and a synthesis request to the meeting device 60.

[0231] S32: The terminal communication unit 61 of the meeting device 60 receives the voice data and the synthesis request, and the voice synthesis unit 65 synthesizes the received voice data with the surrounding voice data collected by the sound collection unit 64. For example, the voice synthesis unit 65 adds the two pieces of voice data together. Because clear voices around the meeting device 60 are recorded, the accuracy of converting voices around the meeting device 60 (conference room side) into text is improved.

[0232] This voice synthesis can also be performed by the terminal device 10. The recording function may be distributed to the meeting device 60, and the voice processing may be distributed to the terminal device 10. In this case, the load on the meeting device 60 can be reduced.

[0233] S33: Furthermore, the panoramic image creating unit 62 of the meeting device 60 creates a panoramic image, and the speaker image creating unit 63 creates a speaker image.

[0234] S34: The device communication unit 16 of the information recording application 41 repeatedly acquires the panoramic image and the speaker image from the meeting device 60. The device communication unit 16 also repeatedly acquires the synthesized audio data from the meeting device 60. These acquisitions may be performed by the device communication unit 16 making a request to the meeting device 60. Alternatively, the meeting device 60 may automatically send the panoramic image and the speaker image upon receiving a notification that the camera toggle button 211 is on. The meeting device 60 may automatically send the synthesized audio data to the information recording application 41 upon receiving a request to synthesize audio data.

[0235] S35: The recording control unit 17 of the information recording application 41 creates a combined image by arranging the application screen, panoramic image, and speaker image acquired from the remote conference application 42. The recording control unit 17 repeatedly creates combined images and creates a combined image video by specifying each combined image as a frame that constitutes the video. The recording control unit 17 also saves the audio data received from the meeting device 60.

[0236] S36: The communication unit 36 ​​of the interactive whiteboard 2 preferably associates object information such as handwritten data for each stroke with the conference ID and transmits the information to the information processing system 50.

[0237] The information recording application 41 repeats the above steps S30 to S36.

[0238] The processing of steps S30 to S36 does not have to be performed in the order shown in FIG. 32, and the synthesis of the audio data and the creation of the combined image may be performed in the reverse order.

[0239] <Major Effects> As described above, in the record information creation system 100 of this embodiment, the meeting device 60 acquires device identification information of devices used in a conference room, and the terminal device 10 transmits this information to the information processing system 50, allowing the information processing system 50 to associate the devices as devices used in the conference. Because the meeting device 60 is connected to the terminal device 10 that communicates with the information processing system 50, the meeting device 60 can also be associated with the conference. There is no need for the user to capture the two-dimensional code displayed by the device with a camera or to register the device and meeting device 60 in the information processing system 50 for the conference, and multiple devices (meeting device 60, electronic whiteboard) can be associated with each other with minimal user effort. [Example]

[0240] In this embodiment, an electronic whiteboard 2 that notifies device identification information by sound and its processing will be described. Note that in this embodiment, the hardware configuration diagrams of Figures 4, 5, and 8 described in the above embodiments can be used.

[0241] <About the function> 33 is an example of a functional block diagram illustrating, by dividing the functions of the terminal device 10, the meeting device 60, and the information processing system 50 in the record information creation system 100. The explanation of FIG. 33 will mainly focus on the differences from FIG. 9.

[0242] 33, the terminal device 10 has a sound data analysis unit 23. As will be described later, the sound data analysis unit 23 performs spectrum analysis on the sound data to detect frequencies contained in the sound data, and converts specific frequencies into bit data to obtain device identification information.

[0243] FIG. 34 is a functional block diagram explaining the functions of the interactive whiteboard 2 by dividing them into blocks. The explanation of FIG. 34 will mainly focus on the differences from FIG. 13. The interactive whiteboard 2 in FIG. 34 has a sound data generation unit 37. The sound data generation unit 37 converts the letters and numbers that make up the device identification information into frequencies of time T, and generates sound signals by sampling at regular intervals, similar to PCM conversion. The sound signals are converted to analog by a D / A converter in the speaker 450 and output from the speaker 450. The sound data generation unit 37 also serves as an output unit that outputs sound signals.

[0244] In this embodiment, the sound collection unit 64 of the meeting device 60 also serves as an acquisition unit that acquires sound data.

[0245] <Sound data generation> Figure 35 shows an example of the sound frequencies and bit patterns used to transmit device identification information. In Figure 35, the frequencies used to transmit device identification information by sound are 18 KHz, 19 KHz, and 20 KHz. 18 KHz is the frequency of a pilot signal added before the message. The length of time T corresponds to 1 bit. Also, 19 kHz corresponds to 0, and 20 kHz corresponds to 1. Therefore, Figure 35(a) shows the value 00, Figure 35(b) shows the value 01, Figure 35(c) shows the value 10, and Figure 35(d) shows the value 11.

[0246] The pilot signal is transmitted by adding 2 bits to every 8 bits of data. If the sound signal represents one alphabet or number using 8 bits, 10 bits are required to transmit the device identification information for one character. For example, if the interactive whiteboard 2 transmits the 8-bit ASCII character "e" (01100101 in binary, 0x65 in hexadecimal), the frequency pattern of the sound signal will be as shown in Figure 36.

[0247] The device identification information of the interactive whiteboard 2 uses the numbers 0 to 9 and the English capital letters A to Z. Therefore, the device identification information is expressed as an 8-bit ASCII code. Each character of the device identification information, together with a pilot signal, has a frequency pattern as shown in FIG. 36. The frequency patterns corresponding to the numbers 0 to 9 and the letters A to Z and a sound signal of one bit's time T are PCM-encoded and pre-stored in the interactive whiteboard 2. The sound data generator 37 uses a sound source to convert the PCM-encoded sound data into digital audio data for each character that constitutes the device identification information of the interactive whiteboard 2.

[0248] The sound collection unit 64 of the meeting device 60 collects ambient sounds, and therefore also collects the sound signal generated by the sound data generation unit 37. The sound data analysis unit 23 of the terminal device 10 performs spectral analysis (Fourier transform) on the sound signal at regular intervals (e.g., every tens of milliseconds) to obtain a spectrum with a peak at 18 kHz, 19 kHz, or 20 kHz. The sound data analysis unit 23 detects the beginning of the characters constituting the device identification information at 18 kHz and converts 19 kHz or 20 kHz to 0 or 1. Additionally, since the time used for spectral analysis (e.g., every tens of milliseconds) is shorter than the time T, the sound data analysis unit 23 aggregates the data into one bit of 0 or 1 for each time T depending on whether there are more 0s or 1s within the time T, and reproduces the device identification information. The time used for spectral analysis (e.g., every tens of milliseconds) may be the same as the time T.

[0249] <Operation procedure> 37 is a sequence diagram illustrating the processing performed by the record information creation system 100. In the explanation of FIG. 37, differences from FIG. 20 will be mainly explained. First, step S41 may be the same as step S1 in FIG.

[0250] S42: On an electronic whiteboard 2 placed in a conference room to be used for a meeting, a user presses a button to output the device identification information of the electronic whiteboard 2. The sound data generator 37 of the electronic whiteboard 2 generates a sound signal representing the device identification information of the electronic whiteboard 2 in terms of frequency, and outputs the sound signal from the speaker. The sound signal may be output automatically when the power is turned on.

[0251] S43: The sound collection unit 64 of the meeting device 60 collects the sound signal output from the interactive whiteboard 2 with a microphone and PCM-encodes the sound signal.

[0252] S44: The terminal communication unit 61 of the meeting device 60 transmits the sound signal to the terminal apparatus 10.

[0253] S45: The device communication unit 16 of the terminal device 10 receives the sound signal, and the sound data analysis unit 23 performs frequency analysis on the sound signal, dividing it into 18 kHz pilot signals, and converts the frequencies (19 kHz, 20 kHz) contained in each divided sound signal into 8-bit bit strings based on the conversion rules in Fig. 35. The sound data analysis unit 23 converts this into device identification information using an 8-bit ASCII code conversion table. This conversion table associates the numbers 0 to 9 and English uppercase letters A to Z (8-bit bit strings) with the frequency patterns shown in Fig. 36.

[0254] S46: The communication unit 11 of the terminal device 10 transmits the device identification information to the information processing system 50. The subsequent processing may be the same as in FIG.

[0255] In FIG. 37, the meeting device 60 collects the sound signal, but the terminal device 10 may directly collect the sound signal.

[0256] In the above embodiment, the sound frequency is set to 18 KHz to 20 KHz, but since this frequency is in the audible range, there is a risk that the sound may be heard by the user. Therefore, it is preferable that the electronic whiteboard 2 outputs the device identification information as sound using ultrasonic waves of, for example, about 50 to 100 KHz.

[0257] Figure 38 shows an example of the correspondence between frequency and bit data in the case of ultrasonic waves. In Figure 38, ultrasonic frequencies of 50 KHz (used for pilot signals), 55 KHz, and 60 KHz are shown, but ultrasonic waves may be of other frequencies. Figure 38(a) shows the value 00, Figure 38(b) shows the value 01, Figure 38(c) shows the value 10, and Figure 38(d) shows the value 11.

[0258] In this case, it is preferable that the speaker 450 of the interactive whiteboard 2 also supports ultrasonic waves, and that the microphone 608 of the meeting device 60 also supports ultrasonic waves.

[0259] <Major Effects> According to this embodiment, in addition to the effects of the first embodiment, the electronic whiteboard 2 can notify the meeting device 60 of the device identification information by sound, so that even if someone is in front of the electronic whiteboard 2, the meeting device 60 can easily obtain the device identification information. [Example]

[0260] In this embodiment, the process at the end of recording (end of the conference) will be described. When the user ends the recording, the information processing system 50 deletes the association between the conference ID and the device identification information (opens the electronic whiteboard 2), the electronic whiteboard 2 displays a message indicating the end of recording, and the display and output of the device identification information is resumed.

[0261] In this embodiment, the hardware configuration diagrams of FIGS. 4, 5, and 8 and the functional block diagram shown in FIG. 9 described in the above embodiments can be used.

[0262] <Action or Processing> FIG. 39 is a sequence diagram illustrating the process performed by the record information creation system 100 when recording of a conference is ended.

[0263] S51: A user is about to end a meeting in which the meeting device 60 and the interactive whiteboard 2 are used in the same meeting. The user presses the recording end button 227 of the information recording application 41. The operation reception unit 12 receives the press.

[0264] S52: The recording control unit 17 of the information recording application 41 stops recording the combined image moving image and stops recording the audio data.

[0265] S53: Furthermore, the communication unit 11 of the information recording application 41 transmits a conference end notification to the information processing system 50, specifying the conference ID.

[0266] S54: The communication unit 51 of the information processing system 50 receives the conference end notification, and the communication management unit 54 transmits the conference end notification to the electronic whiteboard 2 that transmitted the conference ID.

[0267] S55: The communication unit 36 ​​of the electronic whiteboard 2 receives the meeting end notification, and the display control unit 34 displays a meeting end notification screen.

[0268] 40 is an example of a meeting end notification screen 310 displayed by the interactive whiteboard 2. The meeting end notification screen 310 has a message 311 that reads, "The meeting (recording of information) will end. Any handwritten notes from now on will not be saved in the recorded information. Please press End if you wish to end the meeting, or Continue if you wish to continue it," an End button 312 (an example of a display component), and a Continue button 313. When the user presses the End button 312, recording of the objects displayed by the interactive whiteboard 2 ends (object information is not sent to the information processing system 50). When the Continue button 313 is pressed, the user can add to the interactive whiteboard 2.

[0269] S56: The user presses the end button 312. The contact position detection unit 31 receives the press.

[0270] S57: The communication unit 36 ​​of the interactive whiteboard 2 specifies the conference ID and transmits a conference end acknowledgement notice to the information processing system 50. The communication unit 36 ​​ends transmission of the object information to the information processing system 50.

[0271] S58: When the communication unit 51 of the information processing system 50 receives the conference end acknowledgement notification, the device management unit 55 deletes the association (association information) between the conference ID and the device identification information.

[0272] S59: The communication unit 51 of the information processing system 50 transmits a notification of completion of the link cancellation to the interactive whiteboard 2.

[0273] S60: The communication unit 36 ​​of the interactive whiteboard 2 receives the notification of completion of link cancellation and resumes output of the two-dimensional code or barcode in the case of embodiment 1, or the sound signal in the case of embodiment 2. The data recording unit 33 deletes the conference ID.

[0274] S61: The communication unit 51 of the information processing system 50 transmits a cooperation cancellation completion notification to the terminal device 10. The communication unit 11 of the information recording application 41 receives the cooperation cancellation completion notification and deletes the conference ID.

[0275] S62: In response to the notification that the linkage has been terminated, the device communication unit 16 of the information recording application 41 notifies the meeting device 60 that recording has ended. The meeting device 60 continues creating panoramic images and speaker images and synthesizing audio. However, the meeting device 60 may change processing, such as changing the resolution or frame rate, depending on whether recording is in progress. The meeting device 60 may interrupt creating panoramic images and speaker images and synthesizing audio if the information recording application 41 is not operated, for example.

[0276] S63: The recording control unit 17 of the information recording application 41 combines the audio data with the combined image moving image to create a combined image moving image with audio.

[0277] S64: Furthermore, if the user checks the checkbox 215 associated with "Automatically transcribe recording after uploading" on the recording setting screen 210, the audio data processing unit 18 requests the information processing system 50 to convert the audio data into text data. In detail, the audio data processing unit 18 specifies the URL of the save destination via the communication unit 11, and transmits a request to the information processing system 50 to convert the audio data combined into the combined image video together with the conference ID and recording ID.

[0278] S65: The communication unit 51 of the information processing system 50 receives a request to convert the voice data, and the text conversion unit 56 converts the voice data into text data using the voice recognition service system 80. The communication unit 51 saves the text data in the same storage location as the combined image video. Note that in the recording information storage unit 5002, the text data is associated with the combined image video by the conference ID and recording ID. The terminal device 10 may request voice recognition from the voice recognition service system 80 and save the text data obtained from the voice recognition service system 80 in the storage location.

[0279] S66: The upload unit 20 of the information recording application 41 also saves the combined image moving image in the destination for saving the combined image moving image via the communication unit 11. The combined image moving image is associated with the conference ID and the recording ID in the recording information storage unit 5002. The combined image moving image is recorded as having been uploaded.

[0280] S67: The communication unit 51 of the information processing system 50 also associates the object information transmitted from the electronic whiteboard 2 during the conference with the conference ID and saves the object information in the same storage location as the combined image video. Therefore, the object information, the combined image video, and the text data are associated with each other by the conference ID.

[0281] The save destination is notified to the user, so the user can share the combined image video with participants by informing them of the save destination by email, etc. Even if the combined image video, audio data, text data, and object information are created using different devices, they can be collected and stored in a single storage location, allowing users to easily view them later.

[0282] <Major Effects> According to this embodiment, when recording ends (the meeting ends), it becomes possible to delete the association between the meeting ID and the device identification information in the information processing system 50 (opening the electronic whiteboard 2), display the end of recording on the electronic whiteboard 2, and resume displaying and outputting the device identification information, etc. [Example]

[0283] In this embodiment, the process at the end of recording will be described in the same manner as in the third embodiment, but a different ending method will be described.

[0284] In this embodiment, it is assumed that the hardware configuration diagrams of FIGS. 4, 5, and 8 described in the above embodiments can be used.

[0285] <About the function> 41 is an example of a functional block diagram illustrating, by dividing the functions of the terminal device 10, the meeting device 60, and the information processing system 50 in the record information creation system 100. The explanation of FIG. 41 will mainly focus on the differences from FIG. 9.

[0286] 41, the terminal device 10 has an end detection unit 24. The end detection unit 24 stores the shape of the electronic whiteboard 2 and determines whether the electronic whiteboard 2 is detected from the panoramic image. First, when a two-dimensional code or a barcode is detected, the end detection unit 24 detects the shape (circumscribed rectangle) of the electronic whiteboard 2 in the conference room by using pattern matching or machine learning.

[0287] Fig. 42 is an image 241 of the shape of the electronic whiteboard 2 detected from the panoramic image. During recording, the end detection unit 24 repeatedly detects the electronic whiteboard 2 from the panoramic image using Fig. 42 as a standard pattern, and when it determines that the electronic whiteboard 2 does not exist, it determines that the meeting (recording) has ended.

[0288] According to this process, the user can end the conference without pressing the recording stop button 227. The user can also end the conference by blocking the space between the meeting device 60 and the interactive whiteboard 2 with their hand (or by operating the mute button). The user can also end the conference by leaving the conference room while the terminal device 10 and the meeting device 60 are still connected.

[0289] The end detection unit 24 may also detect the end of the conference by detecting the unplugging of the USB cable (wired cable). The device communication unit 16 detects that the USB cable has been unplugged when the external device connection I / F 508 does not detect voltage, and notifies the end detection unit 24. The device communication unit 16 may also detect a communication interruption when there is no response from the meeting device 60. In this case, too, the conference can be ended by the usual operation of unplugging the cable, without the user having to press the recording stop button 227.

[0290] <Action or Processing> 43 is a sequence diagram illustrating the process when the record information creation system 100 ends recording of a conference. The explanation of FIG. 43 will mainly focus on the differences from FIG.

[0291] S70: The terminal communication unit 61 of the meeting device 60 transmits the panoramic image to the terminal device 10.

[0292] S71: The end detection unit 24 detects that the electronic whiteboard 2 is not detected from the panoramic image or that the USB cable has been unplugged, as described above. Note that the meeting device 60 can also detect that the electronic whiteboard 2 is not detected from the panoramic image.

[0293] The subsequent processing may be the same as in FIG.

[0294] <Major Effects> According to this embodiment, in addition to the effect of the third embodiment, the number of user operations can be reduced and the conference (recording) can be ended. [Example]

[0295] In this embodiment, the process at the end of recording will be described in the same manner as in the third embodiment, but a different ending method will be described.

[0296] In this embodiment, it is assumed that the hardware configuration diagrams of FIGS. 4, 5, and 8 described in the above embodiments can be used.

[0297] <About the function> 44 is an example of a functional block diagram illustrating, by dividing the functions of the terminal device 10, the meeting device 60, and the information processing system 50 in the record information creation system 100, the differences from FIG. 33 will be mainly explained in the explanation of FIG.

[0298] 44, the terminal device 10 has an end detection unit 24. The end detection unit 24 detects that the meeting has ended when it determines that the pilot signal is not included in the sound captured by the microphone of the meeting device 60. That is, during the meeting (during recording), the interactive whiteboard 2 intermittently outputs a pilot signal, and the end of the meeting (end of recording) can be detected when the meeting device 60 no longer receives the pilot signal.

[0299] With this processing, the user can end the meeting without pressing the end recording button 227 by simply blocking the space between the meeting device 60 and the electronic whiteboard 2 with their hand, or by leaving the conference room while the terminal device 10 and the meeting device 60 are still connected.

[0300] <Action or Processing> 45 is a sequence diagram illustrating the process when the record information creation system 100 ends recording of a conference. The explanation of FIG. 45 will mainly focus on the differences from FIG.

[0301] S80: The terminal communication unit 61 of the meeting device 60 transmits the sound data to the terminal device 10.

[0302] S81: The end detection unit 24 determines whether or not a pilot signal is included in the sound data transmitted from the meeting device 60 as described above. This determination may be made directly from the sound collected by the information recording application 41, or may receive a non-detection result from the meeting device 60.

[0303] The subsequent processing may be the same as in FIG.

[0304] <Major Effects> According to this embodiment, in addition to the effect of the third embodiment, the number of user operations can be reduced and the conference (recording) can be ended.

[0305] <Other application examples> The best mode for carrying out the present invention has been described above using examples, but the present invention is not limited to these examples in any way, and various modifications and substitutions can be made within the scope that does not deviate from the gist of the present invention.

[0306] For example, the terminal device 10 and the meeting device 60 may be integrated. The meeting device 60 may be externally attached to the terminal device 10. The meeting device 60 may also be a hemispherical camera, a microphone, and a speaker connected by a cable.

[0307] Furthermore, a meeting device 60 may also be placed at another location 101. The other location 101 separately uses the meeting device 60 to create combined image video and text data. Furthermore, multiple meeting devices 60 may be placed at one location. In this case, multiple pieces of record information are created for each meeting device 60.

[0308] The layout of the panoramic image 203, the speaker image 204, and the app screen in the combined image video used in this embodiment is merely an example. The panoramic image 203 may be on the bottom and the speaker image 204 on top, or the user may change the layout, or may individually switch between displaying and hiding the panoramic image 203 and the speaker image 204 during playback.

[0309] Furthermore, the configuration examples in Figure 9 and the like are divided according to main functions to facilitate understanding of the processing by the terminal device 10, the meeting device 60, and the information processing system 50. The method of dividing the processing units and the names thereof do not limit the present invention. The processing by the terminal device 10, the meeting device 60, and the information processing system 50 can be divided into even more processing units depending on the processing content. Furthermore, the processing can also be divided so that one processing unit includes even more processes.

[0310] Additionally, the devices described in the examples are merely illustrative of one of several computing environments for implementing the embodiments disclosed herein. In one embodiment, information processing system 50 includes multiple computing devices, such as a server cluster, configured to communicate with each other via any type of communication link, including a network, shared memory, etc., and to perform the processes disclosed herein.

[0311] Furthermore, the information processing system 50 can be configured to share the disclosed processing steps, such as those shown in FIG. 20, in various combinations. For example, a process executed by a specific unit can be executed by multiple information processing devices included in the information processing system 50. Furthermore, the information processing system 50 may be integrated into a single server device, or may be divided into multiple devices.

[0312] Each function of the above-described embodiments can be realized by one or more processing circuits. Here, the term "processing circuit" in this specification includes a processor programmed to perform each function by software, such as a processor implemented by an electronic circuit, as well as devices such as an ASIC (Application Specific Integrated Circuit), a DSP (Digital Signal Processor), an FPGA (Field Programmable Gate Array), and a conventional circuit module designed to perform each of the above-described functions. [Explanation of symbols]

[0313] 10 Terminal Equipment 50 Information Processing Systems 60 Meeting Devices 100 Recording Information Creation System [Prior art documents] [Patent documents]

[0314] [Patent Document 1] Japanese Patent Application Laid-Open No. 2008-124689

Claims

1. A device management system including a first device that processes information relating to communication, a second device, and an information processing system, The first device is The display and an output unit that outputs image information to the display; The second device is an acquisition unit that acquires the image information output by the first device by capturing the image; a terminal communication unit that transmits information acquired from the image information to the information processing system, The information processing system includes: a communication unit that receives information obtained from the image information transmitted by the second device; a device management unit that associates the first device and the second device with the same communication based on information acquired from the image information; 1. A device management system comprising:

2. A device management system including a first device that processes information relating to communication, a second device, and an information processing system, The first device is The display and an output unit that outputs image information to the display; The second device is an acquisition unit that acquires the image information output by the first device by capturing the image; a terminal communication unit that transmits a registration request for the communication to the information processing system based on the image information, The information processing system includes: a communication unit that receives a registration request for the communication from the second device; a device management unit that associates the first device and the second device with the same communication based on a registration request for the communication; 1. A device management system comprising:

3. The information processing system comprises: a communication management unit that issues identification information of the communication; 3. The device management system according to claim 1, wherein the communication unit transmits the identification information of the communication to the first device and the second device.

4. transmitting object information displayed on the display by the first device in association with the identification information of the communication to the information processing system; the second device generates image data of a surrounding area of ​​the second device in the communication, and transmits the image data of the surrounding area to the information processing system in association with identification information of the communication; An equipment management system according to any one of claims 1 to 3, characterized in that the information processing system receives the object information from the first device and receives the surrounding image data from the second device, and the information processing system stores the object information and the surrounding image data in association with the identification information of the communication.

5. An equipment management system described in any one of claims 1 to 4, characterized in that the image information is a two-dimensional code or a barcode.

6. The device management system according to claim 5 , wherein the output unit displays the two-dimensional code or the barcode above a center line of a display screen of the display of the first device.

7. The device management system according to claim 5 , wherein the output unit displays the center of the two-dimensional code or the barcode above a center line of the display screen of the display of the first device.

8. An equipment management system as described in any one of claims 1 to 7, characterized in that if the acquisition unit does not acquire the image information within a certain period of time, it displays a message indicating that the image information has not been acquired.

9. The second device includes a recording control unit that records the image data of the surroundings; an end detection unit that detects that recording of the surrounding image data is to be ended; transmitting to the information processing system the information that the recording has been terminated, the information being terminated by the termination detection unit; 5. The device management system according to claim 4, wherein the information processing system notifies the first device that recording of the object information has ended.

10. A first device having a display and an output unit that outputs image information to the display, the first device processing information related to communication; and An information processing system that communicates with a second device that processes information related to communication, the second device having an acquisition unit that acquires the image information output by the first device by capturing an image, and a terminal communication unit that transmits information acquired from the image information to an information processing system, a communication unit that receives information obtained from the image information transmitted by the second device; a device management unit that associates the first device and the second device with the same communication based on information acquired from the image information; An information processing system comprising:

11. A first device having a display and an output unit that outputs image information to the display, the first device processing information related to communication; and A device management method performed by an information processing system communicating with a second device that processes information related to communication, the second device having an acquisition unit that acquires the image information output by the first device by capturing an image, and a terminal communication unit that transmits information acquired from the image information to an information processing system, the method comprising: a step of receiving, by a communication unit, information obtained from the image information transmitted by the second device; a step of associating the first device and the second device with the same communication by a device management unit based on information acquired from the image information; 1. A device management method comprising:

12. A first device having a display and an output unit that outputs image information to the display, the first device processing information related to communication; an information processing system that communicates with a second device that processes information related to communication, the second device having an acquisition unit that acquires the image information output by the first device by capturing an image and a terminal communication unit that transmits information acquired from the image information to an information processing system; a communication unit that receives information obtained from the image information transmitted by the second device; a device management unit that associates the first device and the second device with the same communication based on information acquired from the image information; A program to function as a

13. A device for processing information relating to communications, comprising: an acquisition unit that acquires image information output on a display of another device used for communication by capturing the image; a terminal communication unit that transmits information acquired from the image information to the information processing system so that the information processing system associates the other device with the same communication; An apparatus characterized by having:

14. A device for processing information relating to communications, comprising: an acquisition unit that acquires image information output on a display of another device used for communication by capturing the image; a terminal communication unit that transmits a registration request for the communication to the information processing system so that the information processing system associates the other device with the same communication; An apparatus characterized by having:

15. The device management system according to claim 3 , wherein the information processing system stores the identification information of the first device and the identification information of the second device in association with the same communication identification information.

16. the first device transfers information input in the communication to the information processing system; the information processing system receives, from the first device, information input to the first device; 4. The device management system according to claim 3, wherein the information input to the first device is stored in association with the identification information of the communication.

17. the second device forwards collected information regarding the communication to the information processing system; The device management system described in claim 3, characterized in that the information processing system receives information collected by the second device from the second device and stores the received information collected by the second device in association with the communication identification information.

18. 3. The device management system according to claim 1, wherein the image information includes identification information of the first device.

19. 3. The device management system according to claim 1, wherein the information processing system receives information relating to communication from the first device and transfers the information to the second device.

20. The device management system of claim 17 , wherein the information collected by the second device regarding the communication includes video.

21. 3. The device management system according to claim 1, wherein the first device and the second device can input information about the communication to each other via the information processing system during the communication.

Images