System, apparatus, and method

Abstract

To provide an apparatus system for adjusting a sound volume of an appropriate apparatus.SOLUTION: An apparatus system includes a first apparatus including voice input / output means and a second apparatus including voice input means and output means. The first apparatus includes: an acquisition unit for acquiring information on the second apparatus output by the second apparatus; and a terminal communication unit for transmitting the information on the second apparatus acquired by the acquisition unit to a terminal device. The second apparatus includes: an output unit for outputting the information on the second apparatus; and an adjustment unit for adjusting a sound volume of at least one of the voice input means and the output means of the second apparatus according to an adjustment request for a sound volume of the voice to the second apparatus received from the terminal device via a network.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a device system, a volume adjustment method, a second device, and a first device. [Background technology]

[0002] Remote communication systems are known in which images and audio are transmitted in real time from one location to one or more other locations, allowing users in remote locations to hold conferences using images and audio. Remote communication may involve the use of devices such as electronic whiteboards in conference rooms.

[0003] There is known a technique for suppressing feedback caused by multiple devices placed in a conference room (see, for example, Patent Document 1). Patent Document 1 discloses a technique for preventing feedback by muting one of the terminals. Summary of the Invention [Problem to be solved by the invention]

[0004] However, the conventional technology has a problem in that it may mute devices whose volume should not be adjusted. For example, the conventional technology may mute devices based on distance, which may result in silencing devices that should not be muted.

[0005] In view of the above-mentioned problems, an object of the present invention is to provide an equipment system that adjusts the volume of an appropriate equipment. [Means for solving the problem]

[0006] In view of the above problems, the present invention provides a device system including a first device having an audio input / output means and a second device having an audio input means and an audio output means, wherein the first device includes an acquisition unit that acquires information about the second device output by the second device, and a terminal communication unit that transmits the information about the second device acquired by the acquisition unit to a terminal device; The second device has an output unit that outputs information regarding the second device, and an adjustment unit that adjusts the volume of at least one of the audio input means or output means of the second device in response to a request for adjusting the volume of the audio for the second device received from the terminal device via a network. [Effects of the Invention]

[0007] It is possible to provide an equipment system that adjusts the volume of the appropriate equipment. [Brief explanation of the drawings]

[0008] [Figure 1] 10A and 10B are diagrams illustrating an outline of creating record information that saves the screen of an application executed during a remote conference together with a panoramic image of the surrounding area. [Figure 2] 10A and 10B are diagrams illustrating a process in which a terminal device mutes a microphone and a speaker of an interactive whiteboard. [Figure 3] FIG. 1 is a diagram illustrating an example of the configuration of a record information creation system. [Figure 4] FIG. 2 is a diagram illustrating an example of the hardware configuration of an information processing system and a terminal device. [Figure 5] FIG. 2 is a diagram illustrating an example of the hardware configuration of a meeting device. [Figure 6] FIG. 10 is a diagram illustrating the imaging range of the meeting device. [Figure 7] 10A and 10B are diagrams illustrating the cutting out of a panoramic image and a speaker image. [Figure 8] FIG. 2 is a diagram illustrating a hardware configuration of an example of an electronic whiteboard. [Figure 9] 1 is an example of a functional block diagram illustrating functions of a terminal device, a meeting device, and an information processing system in a record information creation system, divided into blocks. [Figure 10] 4 is a diagram showing an example of moving image recording information stored in an information storage unit. FIG. [Figure 11] FIG. 10 is a diagram illustrating an example of conference information managed by a communication management unit. [Figure 12]10 is a diagram showing an example of association information stored in an association information storage unit, in which conference IDs are associated with device identification information; FIG. [Figure 13] FIG. 1 is an example of a functional block diagram illustrating functions of an electronic whiteboard divided into blocks. [Figure 14] 4 is a diagram showing an example of device identification information stored in a device information storage unit; FIG. [Figure 15] FIG. 2 is a diagram illustrating object information stored in an object information storage unit. [Figure 16] FIG. 10 is a sequence diagram showing the flow of processing in which the interactive whiteboard mutes voice input and output in response to participation in a conference (mute procedure 1). [Figure 17] FIG. 10 is a diagram illustrating an example of a conference participation screen displayed by a terminal device. [Figure 18] FIG. 10 is a diagram showing an example of a two-dimensional code displayed on an electronic whiteboard. [Figure 19] 10A and 10B are diagrams illustrating an example of a display method for a two-dimensional code displayed on an electronic whiteboard. [Figure 20] 10A and 10B are diagrams illustrating an example of a display method for a two-dimensional code displayed on an electronic whiteboard. [Figure 21] 10A and 10B are diagrams illustrating an example of a display method for a two-dimensional code displayed on an electronic whiteboard. [Figure 22] 10A and 10B are diagrams illustrating an example of a display method for a two-dimensional code displayed on an electronic whiteboard. [Figure 23] 10A and 10B are diagrams illustrating an example of a display method for a two-dimensional code displayed on an electronic whiteboard. [Figure 24] 10A and 10B are diagrams illustrating an example of a display method for a two-dimensional code displayed on an electronic whiteboard. [Figure 25] 10A and 10B are diagrams illustrating an example of a display method for a two-dimensional code displayed on an interactive whiteboard when the camera of the meeting device is a hemispherical camera. [Figure 26] 10A and 10B are diagrams illustrating an example of a display method of a barcode displayed on an electronic whiteboard. [Figure 27] FIG. 10 is a diagram showing an example of a screen displayed by the interactive whiteboard after muting. [Figure 28]FIG. 10 is a sequence diagram showing the flow of processing in which the interactive whiteboard mutes voice input and output in response to participation in a conference (mute procedure 2). [Figure 29] FIG. 2 is a diagram illustrating an example of a format of an audio signal. [Figure 30] FIG. 10 is a sequence diagram showing the flow of processing in which the interactive whiteboard mutes voice input and output in response to participation in a conference (mute procedure 3). [Figure 31] FIG. 10 is a sequence diagram showing the flow of processing in which the interactive whiteboard mutes voice input and output in response to participation in a conference (mute procedure 4). [Figure 32] FIG. 10 is a diagram illustrating an example of a device detection start screen displayed by the terminal device. [Figure 33] FIG. 10 is a sequence diagram showing the flow of processing in which the interactive whiteboard mutes voice input and output in response to participation in a conference (mute procedure 5). [Figure 34] FIG. 10 is a sequence diagram showing the flow of processing in which the interactive whiteboard mutes voice input and output in response to participation in a conference (mute procedure 6). [Figure 35] FIG. 10 is a sequence diagram showing the flow of processing in which the interactive whiteboard mutes voice input and output in response to participation in a conference (mute procedure 7). [Figure 36] FIG. 10 is a sequence diagram showing the flow of processing in which the interactive whiteboard mutes voice input and output in response to participation in a conference (mute procedure 8). [Figure 37] 10 is a sequence diagram showing an example of a process flow for canceling muting on an electronic whiteboard when a participant leaves a meeting (unmuting 1); [Figure 38] FIG. 10 is a diagram illustrating an example of a conference exit instruction screen displayed on the electronic whiteboard. [Figure 39] FIG. 10 is a sequence diagram illustrating an example of a process flow in which a terminal device unmutes an electronic whiteboard when leaving a conference (unmuting 2); [Figure 40] 10 is a sequence diagram showing an example of a process flow for unmuting the interactive whiteboard when communication between the terminal device and the information processing system is disconnected (unmuting 3); [Figure 41]10 is a sequence diagram illustrating an example of a process of ending display of a two-dimensional code without receiving a mute request after the interactive whiteboard is started. [Figure 42] FIG. 10 is a diagram showing an example of an initial screen displayed by an information recording application running on the terminal device after login. [Figure 43] FIG. 10 is a diagram showing an example of a recording setting screen displayed by an information recording application. [Figure 44] FIG. 10 is a diagram showing an example of a recording screen displayed by the information recording application during recording. [Figure 45] FIG. 10 is a diagram illustrating an example of a conference list screen displayed by the information recording application. [Figure 46] 10 is a sequence diagram illustrating an example of a procedure in which the information recording application records a panoramic image, a speaker image, and an application screen. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, an equipment system and a method for adjusting the volume performed by the equipment system will be described as an example of an embodiment of the present invention.

[0010] <An example of how to create minutes for a remote meeting> First, an outline of a method for creating minutes using a panoramic image and an application screen will be described with reference to Fig. 1. Fig. 1 is a diagram for explaining an outline of creating record information that saves the screen of an application executed during a remote conference together with a panoramic image of the surrounding area. As shown in Fig. 1, a user at a home location 102 is using a remote conference service system 90 to hold a remote conference with another location 101.

[0011] The record information creation system 100 of this embodiment creates record information (such as minutes) using a horizontal panoramic image (hereinafter referred to as a panoramic image) acquired by a meeting device 60 equipped with an imaging means capable of capturing 360° images of the surroundings, a microphone, and a speaker, by processing information captured from the surroundings, and a screen created by an application (hereinafter referred to as an app) executed by the terminal device 10. Regarding audio, the record information creation system 100 synthesizes the audio received by the remote conference app 42 and the audio acquired by the meeting device 60, and includes the synthesized audio in the record information. An outline of this is given below.

[0012] (1) An information recording application 41 and a remote conference application, which will be described later, are running on the terminal device 10. In addition, an application for displaying materials may also be running. The information recording application 41 transmits the audio output by the terminal device 10 (including the audio received by the remote conference application from other locations) to the meeting device 60. The meeting device 60 mixes (combines) the audio it has acquired with the audio from the remote conference application.

[0013] (2) The meeting device 60 is equipped with a microphone and performs processing to extract the speaker from the panoramic image based on the direction from which the voice was captured, thereby creating a speaker image. The meeting device 60 transmits both the panoramic image and the speaker image to the terminal device 10.

[0014] (3) The information recording application 41 running on the terminal device 10 can display the panoramic image 203 and the speaker image 204. The information recording application 41 combines the panoramic image 203 and the speaker image 204 with any application screen selected by the user (for example, the screen 103 of a teleconferencing application). For example, the information recording application 41 combines the panoramic image 203, the speaker image 204, and the application screen 103 so that the panoramic image 203 and the speaker image 204 are arranged on the left and the teleconferencing application screen 103 is arranged on the right (hereinafter referred to as a combined image 105). The application screen is an example of screen information (described later) displayed by each application, such as a teleconferencing application. Since the process of (3) is executed repeatedly, the combined image 105 becomes a moving image (hereinafter referred to as a combined image moving image). The information recording application 41 also combines the synthesized audio with the combined image moving image to create a moving image with audio.

[0015] In this embodiment, an example in which the panoramic image 203, the speaker image 204, and the application screen 103 are combined will be described, but the information recording application 41 may store these separately and arrange them on the screen during playback.

[0016] (4) The information recording application 41 accepts editing operations (cutting out unnecessary parts by the user) and completes the combined image video. The combined image video constitutes part of the recorded information.

[0017] (5) The information recording application 41 transmits the created combined image video (with audio) to the storage service system 70 and stores it there.

[0018] (6) The information recording application 41 also extracts only the audio from the combined image video (the audio before combining may be kept), and transmits the extracted audio to the information processing system 50. The information processing system 50 transmits the audio to the speech recognition service system 80, which converts the audio into text data, and converts the audio into text. The text data also includes data on how many minutes after the start of recording that the speech occurred.

[0019] In the case of real-time text conversion, the meeting device 60 directly transmits the voice to the information processing system 50. The meeting device 60 transmits the text data to the information recording application 41 in real time.

[0020] (7) The information processing system 50 additionally stores the text data in the storage service system 70 that stores the combined image video. The text data constitutes part of the recorded information.

[0021] The information processing system 50 can perform billing processing for the user according to the service used. For example, the billing is calculated based on the amount of text data, the file size of the combined image video, the processing time, etc.

[0022] In this way, the combined image video displays a panoramic image of the surroundings including the user and an image of the speaker, and further displays the screen of an app that was displayed during the remote conference, such as the remote conference app 42. When participants in the remote conference or non-participants view the combined image video as minutes, the situation during the remote conference is reproduced with a sense of realism.

[0023] <Processing Overview> Next, referring to FIG. 2, a process in which the terminal device 10 mutes the interactive whiteboard 2 will be described. FIG. 2 is a diagram illustrating a process in which the terminal device 10 mutes the microphone and speaker of the interactive whiteboard 2. The meeting device 60 and the interactive whiteboard 2 each have a microphone and a speaker. The audio output by the meeting device 60 is collected by the microphone of the interactive whiteboard 2, and the interactive whiteboard 2 transmits the audio to the teleconference service system. The teleconference service system 90 transmits the audio to the terminal device 10, and the meeting device 60 outputs it. If this cycle is repeated, feedback may occur. Conversely, the audio output by the interactive whiteboard 2 is collected by the microphone of the meeting device 60, and the terminal device 10 transmits it to the teleconference service system 90. The teleconference service system 90 transmits the audio to the interactive whiteboard 2, and the electronic whiteboard 2 outputs it. If this cycle is repeated, feedback may occur.

[0024] Therefore, the meeting device 60 of this embodiment suppresses howling in the following manner. (1) A user operates the electronic whiteboard 2 to instruct participation in the meeting. (2) The interactive whiteboard 2 displays a two-dimensional code 8 such as a QR code (registered trademark). The two-dimensional code 8 includes information about the interactive whiteboard 2, such as the IP address and device identification information of the interactive whiteboard 2. (3) The meeting device 60 captures the two-dimensional code and decodes it to obtain the IP address and device identification information of the interactive whiteboard 2. (4) The terminal device 10 requests muting from the electronic whiteboard 2 by calling the API (Application Programming Interface) of the electronic whiteboard 2 using the IP address as the destination. Alternatively, the terminal device 10 requests muting from the electronic whiteboard 2 via the information processing system 50 by specifying device identification information.

[0025] In this way, the meeting device 60 of this embodiment mutes the interactive whiteboard 2 displaying the two-dimensional code, so the correct device can be muted without mistaking the device. Furthermore, a user may want to output audio from the meeting device 60 located at the center of the meeting participants and mute the microphone that collects sound from the interactive whiteboard 2, but such control was not possible with conventional technology. As described above, this embodiment prevents the device from being misplaced.

[0026] <Terminology> Volume adjustment mainly refers to lowering the volume. Volume adjustment refers to adjusting at least one of the volume of sound picked up by a microphone and the volume of sound generated by a speaker. One type of adjustment is muting. Volume adjustment may include controlling the volume to a low level that cannot be detected by a microphone. In this embodiment, volume adjustment is simply described as muting.

[0027] The information input means may be any means for inputting images, sounds, radio waves, etc., and examples thereof include a camera (image capture means), a microphone, and short-distance wireless communication.

[0028] The information output means may be any means for displaying images, outputting sound, radio waves, etc., and may be, for example, a display, a projector, a speaker, or a short-distance wireless communication device.

[0029] The information about the second device is information that can identify the second device in communication with the second device. The information about the second device is, for example, device identification information or an IP address. The information about the second device may include whether the interactive whiteboard is currently muted.

[0030] The image information of the surroundings of the meeting device acquired by the meeting device is image information acquired by the meeting device by capturing an image of the surrounding space (for example, a space of 180° to 360° horizontally) surrounding the meeting device, and refers to an image acquired by performing predetermined processing on the curved image information captured by the meeting device. The predetermined processing is various processing for creating surrounding image information from the captured information, such as flattening the captured curved image. In addition to processing for creating a surrounding image, the predetermined processing may also include processing for cutting out a speaker image and combining surrounding images and speaker images. In this embodiment, the surrounding image is described by the term panoramic image. A panoramic image is an image with a field of view of approximately 180° to 360° horizontally. A panoramic image does not have to be captured by a single meeting device; multiple imaging devices with normal fields of view may be combined. The meeting device is intended to be installed on a table or other suitable location for use in meetings at a base or to grasp the surrounding situation, but it may also be used for monitoring (security, disaster prevention, etc.), watching over (childcare, elderly care, etc.), or analyzing the situation on-site (solutions, marketing, etc.).

[0031] Recorded information is information recorded by the information recording application 41, and is information that is stored and saved so that it can be viewed as information linked to the identification information of a particular conference (meeting), and includes, for example, the following information: Video information created based on the screen information displayed by the selected app (such as a remote conferencing app) and image information of the device's surroundings acquired by the device. During a meeting, audio information is acquired and synthesized between the remote conference app (terminal device) and the meeting device at the base. -Text information converted from the captured audio. Other data and images related to the meeting, such as document files used during the meeting, notes added, translations of text data, and images and stroke data created using the cloud electronic whiteboard service during the meeting. For example, when the information recording application 41 records the screen of a remote conference application or the state of a conference at a base, the recorded information may be minutes of the conference that was held. Minutes are an example of recorded information, and the name of the recorded information changes depending on the content of the remote conference or the base, and may be called, for example, a record of communication or a record of the base situation. Furthermore, the recorded information includes files in multiple formats, such as video files (e.g., combined image videos), audio files, text data (text data obtained by voice recognition), document files, image files, and table-format files, and since the files are associated with each other based on the identification information of the conference, they can be viewed collectively or selectively in chronological order.

[0032] A tenant is a group of users (such as a company, a local government, or some of these organizations) who have signed a contract to receive services from a service provider. In this embodiment, the creation of recorded information and conversion to text data are performed because the tenant has signed a contract with the service provider.

[0033] Remote communication refers to communication through audio and video using software and terminal devices with people at physically distant locations. An example of remote communication is a remote conference, which may also be called a meeting, a conference, a consultation, a contract application, a gathering, a get-together, a seminar, a training session, a study group, a seminar, or a workshop.

[0034] A base is a place where activities are based. An example of a base is a conference room. A conference room is a room set up primarily for meetings. A base can also be a home, a reception desk, a store, a warehouse, an outdoor site, or any other location, as long as it is a place or space where terminal equipment, devices, etc. can be installed.

[0035] Although voice refers to speech sounds uttered by humans, surrounding sounds, etc., and voice data refers to voice converted into data, in the present embodiment, a strict distinction will not be made between them.

[0036] The first device is a device that processes information related to communication, and in this embodiment, is referred to as a meeting device 60. The first device may have an imaging device that captures images of the surroundings, a microphone that captures audio, and a function to receive beacons. The first device may also have a speaker.

[0037] The second device may be any device that displays information. In this embodiment, the second device is described using the term "electronic whiteboard." The electronic whiteboard may also be called an electronic whiteboard, an electronic information board, or the like. A projector is known as a device equivalent to the electronic whiteboard 2. Alternatively, the first device may be a digital signage, a television, a display, a multifunction printer, a video conference terminal, or the like.

[0038] <System configuration example> Next, the system configuration of the record information creation system 100 will be described with reference to Fig. 3. Fig. 3 shows an example of the configuration of the record information creation system 100. Fig. 3 shows one location (home location 102) among multiple locations where a remote conference is held, and a terminal device 10 at the home location 102 communicates with an information processing system 50, a storage service system 70, and a remote conference service system 90 via a network. A meeting device 60 and an electronic whiteboard 2 are also disposed at the home location 102, and the terminal device 10 is communicatively connected to the meeting device 60 via a USB cable or the like. The meeting device 60, the electronic whiteboard 2, and the information processing system 50 all operate as a device system.

[0039] At least an information recording application 41 and a remote conference application 42 run on the terminal device 10. The remote conference application 42 can communicate with terminal devices 10 at other locations 101 via a remote conference service system 90 on the network, allowing users at each location to hold conferences from remote locations. The information recording application 41 creates record information for the remote conferences held by the remote conference application 42 using the functions of the information processing system 50 and the meeting device 60.

[0040] In this embodiment, an example of creating recorded information during a remote conference will be described, but the conference does not have to be a conference involving communication with a remote location. In other words, the conference may be a conference in which only participants from one location participate. In this case, the images captured by the meeting device 60 and the collected audio are saved without being combined, and the processing of the information recording application 41 remains unchanged.

[0041] The terminal device 10 is equipped with a built-in camera with a normal angle of view (or it may be external), which captures a front image including the user 107 operating the terminal device 10. A normal angle of view is an image that is not a panoramic image, and in this embodiment, it is a planar image that is not curved, such as a spherical image. Therefore, the user can hold a conventional teleconference using the teleconference application 42 without being aware of the information recording application 41. The information recording application 41 and the meeting device 60 do not affect the teleconference application 42, except for increasing the processing load on the terminal device 10. The teleconference application 42 can also transmit the panoramic image and speaker images captured by the meeting device 60 to the teleconference service system 90.

[0042] The information recording application 41 communicates with the meeting device 60 to create recorded information. The information recording application 41 also performs tasks such as synthesizing audio captured by the meeting device 60 with audio received from other locations by the remote conference application 42. The meeting device 60 is a meeting device equipped with a panoramic image capture device, a microphone, and a speaker. While the camera of the terminal device 10 can only capture a limited range in front of it, the meeting device 60 can capture the entire surroundings (not necessarily the entire periphery) surrounding the meeting device 60. The meeting device 60 can always fit multiple participants 106 shown in FIG. 3 within its field of view.

[0043] In addition, the meeting device 60 performs operations such as cutting out a speaker image from a panoramic image. The meeting device 60 is not limited to being placed on a desk, but may be placed anywhere in the own base 102. Since the meeting device 60 can capture a celestial sphere image, it may be placed on the ceiling, for example.

[0044] The information recording application 41 displays a list of applications running on the terminal device 10, synthesizes images for the recorded information (creates a combined image video), plays back the combined image video, and accepts editing requests. The information recording application 41 also displays a list of teleconferences that have been held or are scheduled to be held. The list of teleconferences is used for information related to the recorded information, allowing the user to associate the teleconferences with the recorded information.

[0045] The remote conference application 42 is an application that enables a terminal device to remotely communicate with other terminal devices by establishing a communication connection with other terminal devices at other locations 101, transmitting and receiving images and audio, displaying images and outputting audio, etc. The remote conference application can also be called a remote communication application, a remote information common application, etc.

[0046] The information recording application 41 and the remote conference application 42 may be web applications or native applications. A web application is an application that performs processing through collaboration between a program on a web server and a program on a web browser or a native application, and does not need to be installed on the terminal device 10. A native application is an application that is installed on the terminal device 10 and used. In this embodiment, both applications will be described as native applications.

[0047] The terminal device 10 may be a general-purpose information processing device with a communication function, such as a PC (Personal Computer), a smartphone, or a tablet terminal. The terminal device 10 may also be an electronic whiteboard 2, a game console, a PDA (Personal Digital Assistant), a wearable PC, a car navigation system, industrial machinery, medical equipment, or a network home appliance. The terminal device 10 may be any device on which at least the information recording application 41 and the remote conference application 42 can run.

[0048] The electronic whiteboard 2 displays data handwritten on a touch panel using an input means such as a pen or a finger. The electronic whiteboard 2 can communicate with a terminal device 10 or the like via wired or wireless communication, and can capture screens displayed by the terminal device 10 and display them on the display. The electronic whiteboard 2 can convert handwritten data into text data and share information displayed on the display with electronic whiteboards 2 at other locations. The electronic whiteboard 2 may be a simple whiteboard without a touch panel, in which an image is projected by a projector. The electronic whiteboard 2 may also be a tablet terminal, notebook PC (Personal Computer), PDA (Personal Digital Assistant), game console, or the like, equipped with a touch panel.

[0049] The interactive whiteboard 2 can communicate with the information processing system 50. After the interactive whiteboard 2 is powered on, for example, it can receive information from the information processing system 50 by polling the information processing system 50 or by using bidirectional communication such as a Web socket.

[0050] The information processing system 50 is one or more information processing devices arranged on a network. The information processing system 50 has one or more server applications that perform processing in cooperation with the information recording application 41, and a platform service. The server applications manage remote conference lists, manage recorded information recorded in remote conferences, and manage various settings and storage paths. The platform service handles user authentication, contracts, billing, etc.

[0051] All or part of the functions of the information processing system 50 may exist in a cloud environment or an on-premise environment. The information processing system 50 may be configured with multiple server devices or may be configured with a single information processing device. For example, the server application and the infrastructure service may be provided by separate information processing devices, and each function within the server application may have its own information processing device. The information processing system 50 may be integrated with the storage service system 70 and the voice recognition service system 80 described below.

[0052] The storage service system 70 is a storage means on a network and provides a storage service that accepts the saving of files, etc. Known examples of the storage service system 70 include OneDrive (registered trademark), Google Workspace (registered trademark), and DropBox (registered trademark). The storage service system 70 may also be an on-premise NAS (Network Attached Storage), etc.

[0053] The voice recognition service system 80 provides a service that performs voice recognition on voice data and converts it into text data. The voice recognition service system 80 may be a general-purpose commercial service, or may be part of the functions of the information processing system 50. Furthermore, the voice recognition service system 80 may be set up and used as a different service system for each user, tenant, or conference.

[0054] <Hardware configuration example> The hardware configuration of the information processing system 50 and the terminal device 10 according to this embodiment will be described with reference to FIG.

[0055] <<Information processing system and terminal device>> Fig. 4 is a diagram showing an example of the hardware configuration of an information processing system 50 and a terminal device 10 according to this embodiment. As shown in Fig. 4, the information processing system 50 and the terminal device 10 are constructed by a computer, and include a CPU 501, a ROM 502, a RAM 503, a HD (Hard Disk) 504, an HDD (Hard Disk Drive) controller 505, a display 506, an external device connection I / F (Interface) 508, a network I / F 509, a bus line 510, a keyboard 511, a pointing device 512, an optical drive 514, and a media I / F 516.

[0056] Of these, the CPU 501 controls the overall operation of the information processing system 50 and the terminal device 10. The ROM 502 stores programs, such as an IPL, used to drive the CPU 501. The RAM 503 is used as a work area for the CPU 501. The HD 504 stores various data, such as programs. The HDD controller 505 controls the reading and writing of various data from and to the HD 504 under the control of the CPU 501. The display 506 displays various information, such as a cursor, menus, windows, characters, or images. The external device connection I / F 508 is an interface for connecting various external devices. In this case, external devices include, for example, a USB (Universal Serial Bus) memory or a printer. The network I / F 509 is an interface for data communication using a network. The bus line 510 is an address bus, a data bus, or the like, for electrically connecting the components, such as the CPU 501, shown in FIG. 4.

[0057] The keyboard 511 is a type of input means having multiple keys used to input characters, numbers, various instructions, etc. The pointing device 512 is a type of input means for selecting and executing various instructions, selecting a processing target, moving a cursor, etc. The optical drive 514 controls reading and writing of various data from an optical storage medium 513, which is an example of a removable storage medium. The optical storage medium 513 may be a CD, a DVD, Blu-ray (registered trademark), etc. The media I / F 516 controls reading and writing (storing) of data from a storage medium 515, such as a flash memory.

[0058] <<Meeting Devices>> The hardware configuration of the meeting device 60 will be described using Figure 5. Figure 5 is an example of a hardware configuration diagram of a meeting device 60 capable of capturing 360-degree video. In the following, the meeting device 60 is described as a device that uses an imaging element to capture 360-degree video around the device at a predetermined height, but the imaging element may be one, two, or any number of elements. Furthermore, it does not necessarily have to be a dedicated device; a PC, digital camera, smartphone, etc. may have substantially the same functionality by attaching a 360-degree video imaging unit to it after the fact.

[0059] As shown in FIG. 5, the meeting device 60 has an imaging unit 601, an image processing unit 604, an imaging control unit 605, a microphone 608, a sound processing unit 609, a CPU (Central Processing Unit) 611, a ROM (Read Only Memory) 612, an SRAM (Static Random Access Memory) 613, a DRAM (Dynamic Random Access Memory) 614, an operation unit 615, an external device connection I / F 616, a communication unit 617, an antenna 617a, an audio sensor 618, and a speaker 619.

[0060] Of these, the imaging unit 601 includes a wide-angle lens (so-called fisheye lens) 602 with a 360° angle of view for forming a hemispherical image, and imaging elements 603 (image sensors) provided corresponding to each wide-angle lens. The imaging elements 603 include an image sensor such as a CMOS (Complementary Metal Oxide Semiconductor) sensor or a CCD (Charge Coupled Device) sensor that converts the optical image captured by the fisheye lens 602 into image data in the form of an electrical signal and outputs the image data, a timing generation circuit that generates horizontal or vertical synchronization signals and pixel clocks for the image sensor, and a group of registers that set various commands and parameters required for the operation of the imaging elements. The imaging unit 601 may also be a 360° camera, which is an example of imaging means capable of capturing images of the 360° surrounding the meeting device 60.

[0061] The imaging elements 603 (image sensors) of the imaging unit 601 are each connected to the image processing unit 604 via a parallel I / F bus. On the other hand, the imaging elements 603 of the imaging unit 601 are connected to the imaging control unit 605 via a serial I / F bus (such as an I2C bus). The image processing unit 604, the imaging control unit 605, and the sound processing unit 609 are connected to a CPU 611 via a bus 610. Furthermore, a ROM 612, an SRAM 613, a DRAM 614, an operation unit 615, an external device connection I / F 616, a communication unit 617, an audio sensor 618, a speaker 619, and the like are also connected to the bus 610.

[0062] The image processing unit 604 takes in the image data output from the image sensor 603 via a parallel I / F bus, performs predetermined processing on the image data, and creates data for a panoramic image and a speaker image from the fisheye video. Furthermore, the image processing unit 604 synthesizes the panoramic image and the speaker image, etc., and outputs a single video.

[0063] The imaging control unit 605 generally uses the I2C bus to set commands and the like in the registers of the imaging element 603, with the imaging control unit 605 acting as a master device and the imaging element 603 acting as a slave device. Necessary commands and the like are received from the CPU 611. The imaging control unit 605 also uses the I2C bus to retrieve status data and the like from the registers of the imaging element 603 and send it to the CPU 611.

[0064] Furthermore, the imaging control unit 605 instructs the imaging element 603 to output image data when the imaging start button on the operation unit 615 is pressed or when an imaging start instruction is received from the PC. Some meeting devices 60 have a preview display function or a function that supports video display on a display (for example, a PC or smartphone display). In this case, the image data is output from the imaging element 603 continuously at a predetermined frame rate (frames / second).

[0065] As will be described later, the imaging control unit 605 also functions as a synchronization control means that cooperates with the CPU 611 to synchronize the output timing of image data from the imaging element 603. In this embodiment, the meeting device 60 is not provided with a display, but a display unit may be provided.

[0066] Microphone 608 converts sound into sound (signal) data. Sound processing unit 609 takes in the audio data output from microphones 608a, 608b, and 608c via the I / F bus, mixes this audio data, and performs predetermined processing. Sound processing unit 609 also determines the direction of the sound source (speaker) from the audio level (volume) input from microphones 608a, 608b, and 608c.

[0067] The speaker 619 converts input audio data into audio. The microphone 608 and the speaker 619 are examples of input / output means. The input / output means do not need to be integrated into a single configuration, and the input means and the output means may be configured separately.

[0068] The CPU 611 controls the overall operation of the meeting device 60 and executes necessary processing. The ROM 612 stores various programs for operating the meeting device 60. The SRAM 613 and DRAM 614 are work memories that store programs executed by the CPU 611, data in the middle of processing, etc. In particular, the DRAM 614 stores image data in the middle of processing by the image processing unit 604 and data of processed equirectangular projection images.

[0069] The operation unit 615 is a general term for operation buttons such as an imaging start button 615a. By operating the operation unit 615, the user can start imaging or recording, as well as turn the power on / off, establish communication connections, and input settings such as various imaging modes and imaging conditions.

[0070] The external device connection I / F 616 is an interface for connecting various external devices. In this case, the external device may be, for example, a PC (Personal Computer), a display, a projector, an electronic whiteboard, etc. The external device connection I / F 616 may include, for example, a USB terminal, an HDMI (registered trademark) terminal, etc. The video data and image data stored in the DRAM 614 are transmitted to an external terminal via the external device connection I / F 616 or recorded on external media. Furthermore, by using multiple external device connection I / Fs 616, for example, image information captured by the meeting device 60 may be transmitted to a PC via USB for recording, while video (e.g., screen information displayed by a remote conference app) may be acquired from the PC to the meeting device 60, and the video may then be transmitted from the meeting device 60 via HDMI to another external device (e.g., a display, a projector, an electronic whiteboard) for display.

[0071] The communication unit 617 may communicate with a cloud server via the Internet using a wireless communication technology such as Wi-Fi via an antenna 617a provided in the meeting device 60, and transmit the stored video data and image data to the cloud server. The communication unit 617 may also be capable of communicating with nearby devices using a short-range wireless communication technology such as BLE (Bluetooth Low Energy, registered trademark) or NFC.

[0072] The audio sensor 618 is a sensor that acquires 360° audio information to identify from which direction a loud audio is being input within 360° around (horizontal plane) the meeting device 60. The audio processing unit 609 identifies the strongest direction based on the input 360° audio parameters and outputs the audio input direction within 360°.

[0073] Note that other sensors (such as a direction / acceleration sensor or GPS) may calculate direction, position, angle, acceleration, etc., and use this information for image correction or adding location information.

[0074] The image processing unit 604 also creates a panoramic image in the following way: The CPU 611 performs predetermined camera image processing such as Bayer conversion (RGB interpolation processing) on the RAW data input from the image sensor that inputs the spherical image to create a fisheye image (curved image). It then performs DeWarp processing (distortion correction processing) on the created fisheye image (curved image) to create a panoramic image (flat image) that captures a 360-degree view of the area around the meeting device 60.

[0075] The CPU 611 creates speaker images in the following manner. The CPU 611 creates speaker images by cutting out the speaker from a panoramic image (flat image) that captures 360° of the surrounding area. The CPU 611 cuts out the speaker image from the panoramic image, using the voice input direction identified from the 360° output using the audio sensor 618 and the sound processing unit 609 as the speaker's direction. In this case, the method for cutting out a person's image from the voice input direction is to cut out a 30° area centered on the identified voice direction from the 360°, and perform face detection within that area to cut out the image. The CPU 611 further identifies the speaker images for a specific number of people (e.g., three people) who have spoken most recently from the cut-out speaker images.

[0076] The panoramic image and the one or more speaker images may be sent separately to the information recording application 41, or the meeting device 60 may create a single image from them and send it to the information recording application 41. In this embodiment, the panoramic image and the one or more speaker images are sent separately from the meeting device 60 to the information recording application 41.

[0077] Figure 6 is a diagram illustrating the imaging range of the meeting device 60. As shown in Figure 6(a), the meeting device 60 captures an image in a 360° range in the horizontal direction. As shown in Figure 6(b), the imaging range of the meeting device 60 is 0° in the horizontal direction at the height of the meeting device 60, and a predetermined angle above and below.

[0078] FIG. 7 is a diagram illustrating the cutting out of a panoramic image and a speaker image. As shown in FIG. 7, the image captured by the meeting device 60 forms a portion 110 of a sphere and therefore has a three-dimensional shape. As shown in FIG. 6(b), the meeting device 60 performs perspective projection transformation by dividing the angle of view into predetermined angles in the vertical direction and the horizontal direction. By performing perspective projection transformation across the entire 360° horizontal direction without gaps, a predetermined number of planar images are obtained, and a panoramic image 111 is obtained by connecting the predetermined number of planar images horizontally. In addition, the meeting device 60 performs face detection within a predetermined range from the panoramic image centered on the direction of the voice, and creates a speaker image 112 by cutting out an image 15° left and right from the center of the face (30° overall).

[0079] <<Electronic Whiteboard>> Fig. 8 is a hardware configuration diagram of the electronic whiteboard 2. As shown in Fig. 8, the electronic whiteboard 2 includes a CPU (Central Processing Unit) 401, a ROM (Read Only Memory) 402, a RAM (Random Access Memory) 403, an SSD (Solid State Drive) 404, a network I / F 405, and an external device connection I / F (Interface) 406.

[0080] Of these, the CPU 401 controls the overall operation of the electronic whiteboard 2. The ROM 402 stores programs used to start up the OS, such as an IPL (Initial Program Loader). The RAM 403 is used as a work area for the CPU 401. The SSD 404 stores various data, such as programs for the electronic whiteboard 2. The network I / F 405 controls communication with a communication network. The external device connection I / F 406 is an interface for connecting various external devices. In this case, the external devices are, for example, a USB (Universal Serial Bus) memory 430, external devices (a microphone 440 (an example of an input means), a speaker 450 (an example of an output means), and a camera 460).

[0081] The electronic whiteboard 2 also includes a capture device 411, a GPU 412, a display controller 413, a contact sensor 414, a sensor controller 415, an electronic pen controller 416, a short-range communication circuit 419, an antenna 419a of the short-range communication circuit 419, a power switch 422, and selection switches 423.

[0082] Of these, the capture device 411 displays display information on the display of an external PC (Personal Computer) 470 as still images or moving images. The GPU (Graphics Processing Unit) 412 is a semiconductor chip that specializes in graphics. The display controller 413 controls and manages the screen display to output an image output from the GPU 412 to a display 480 or the like. The contact sensor 414 detects contact of the electronic pen 490, the user's hand H, or the like on the display 480. The sensor controller 415 controls the processing of the contact sensor 414. The contact sensor 414 inputs and detects coordinates using an infrared blocking method. This coordinate input and coordinate detection method involves two light receiving and emitting devices installed at both ends of the upper side of the display 480 emitting multiple infrared rays parallel to the display 480, which are reflected by reflecting members installed around the display 480 and return along the same optical path as the light emitted by the light receiving element. The light receiving and emitting device serving as the contact sensor 414 outputs the position (position on the light receiving element) where the infrared light is blocked by an object to the sensor controller 415, and the sensor controller 415 identifies the coordinate position, which is the contact position of the object, from these two pieces of position information. The electronic pen controller 416 determines whether the pen tip or the pen tail has touched the display 480 by communicating with the electronic pen 490 via Bluetooth. The short-range communication circuit 419 is a communication circuit such as NFC (Near Field Communication) or Bluetooth (registered trademark). The power switch 422 is a switch for switching the power of the electronic whiteboard 2 on and off. The selection switches 423 are a group of switches for adjusting, for example, the brightness and color of the display 480.

[0083] Furthermore, the electronic whiteboard 2 includes a bus line 410. The bus line 410 is an address bus, a data bus, or the like for electrically connecting the components such as the CPU 401 shown in FIG.

[0084] The contact sensor 414 is not limited to an infrared blocking type, and may be a capacitive touch panel that identifies the contact position by detecting changes in capacitance. The contact sensor 414 may be a resistive film type touch panel that identifies the contact position by voltage changes across two opposing resistive films. The contact sensor 414 may be an electromagnetic induction type touch panel that identifies the contact position by detecting electromagnetic induction caused by an object touching the display unit. In addition, the contact sensor 414 may use various detection means. Furthermore, the electronic pen controller 416 may determine whether or not the part of the electronic pen 490 that the user grips or other parts of the electronic pen are touched, in addition to the pen tip and pen butt.

[0085] <About the function> Next, the functional configuration of the record information creation system 100 will be described with reference to Fig. 9. Fig. 9 is an example of a functional block diagram that explains the functions of the terminal device 10, the meeting device 60, and the information processing system 50 in the record information creation system 100 by dividing them into blocks.

[0086] <<Terminal Device>> The information recording application 41 running on the terminal device 10 includes a communication unit 11, an operation reception unit 12, a display control unit 13, an application screen acquisition unit 14, an audio acquisition unit 15, a device communication unit 16, a recording control unit 17, an audio data processing unit 18, a recording / playback unit 19, an upload unit 20, an edit processing unit 21, a code analysis unit 22, a mute request unit 23, an audio data analysis unit 24, and a beacon data analysis unit 25. Each of these units included in the terminal device 10 is a function or a means for performing a function realized by one of the components shown in FIG. 4 operating in response to an instruction from the CPU 501 in accordance with the information recording application 41 loaded from the HD 504 to the RAM 503. The terminal device 10 also includes a storage unit 1000 implemented by the HD 504 and the like shown in FIG. 4. The storage unit 1000 includes an information storage unit 1001.

[0087] The communication unit 11 communicates various types of information with the information processing system 50 via a network. The operation reception unit 12 receives various operations for the information recording application 41. The display control unit 13 displays various screens that serve as a user interface in the information recording application 41 in accordance with screen transitions set in the information recording application 41.

[0088] The application screen acquisition unit 14 acquires screen information displayed by an application selected by the user or screen information of the desktop screen from an operating system (OS). If the application selected by the user is a remote conference application 42, screens generated by the remote conference application 42 (images captured by the terminal device cameras at each location, images displayed with shared materials, participant icons, participant names, etc.) are acquired. The screen information displayed by the application (application screen) is information displayed as a window by the running application and acquired as an image by the information recording application. The application window is displayed on a monitor or the like, with the window area depicted as an area within the entire desktop image. The screen information displayed by the application can be acquired by other applications (such as an information recording application) as an image file or a video file consisting of multiple consecutive images via the API of the operating system (OS) or the API of the application that displays it. The screen information of the desktop screen is information composed of an image of the desktop screen generated by the OS and can be similarly acquired as an image file or a video file via the API of the OS. These image files may be in bitmap, PNG, or other formats. The video file format may be MP4 or other formats.

[0089] The audio acquisition unit 15 acquires audio data received by the terminal device 10 from the teleconference application 42 during the teleconference. Note that the audio acquired by the audio acquisition unit 15 does not include audio collected by the terminal device 10, but only audio data received during the teleconference. This is because the meeting device 60 separately collects audio.

[0090] The device communication unit 16 communicates with the meeting device 60 using a USB cable or the like. The device communication unit 16 may communicate with the meeting device 60 via wireless LAN, Bluetooth (registered trademark), or the like. The device communication unit 16 receives the panoramic image and the speaker image from the meeting device 60, and transmits the audio data acquired by the audio acquisition unit 15 to the meeting device 60. The device communication unit 16 receives the audio data synthesized by the meeting device 60.

[0091] The recording control unit 17 combines the panoramic image and speaker image received by the device communication unit 16 and the app screen acquired by the app screen acquisition unit 14 to create a combined image. The recording control unit 17 also chronologically connects the repeatedly created combined images to create a combined image video, and combines the synthesized audio data with the combined image video to create a combined image video with audio. The combination of the panoramic image and speaker image may be performed by the meeting device 60. Also, videos consisting of each image, such as the panoramic image, speaker image, app screen, and image consisting of the panoramic image and speaker image, may be stored in the storage service system 70 as separate video files. In this case, the panoramic video, speaker video, video of the app screen, and combined video of the panoramic image and speaker image may be called up and displayed on a single display screen when viewed.

[0092] The audio data processing unit 18 requests the information processing system 50 to convert the audio data combined with the combined image video extracted by the recording control unit 17, or the synthesized audio data received from the meeting device 60, into text data.

[0093] The recording / playback unit 19 plays back the combined image moving image. The combined image moving image is stored in the terminal device 10 while being recorded, and is then uploaded to the information processing system 50.

[0094] When the remote conference ends, the upload unit 20 transmits the combined image moving image to the information processing system 50.

[0095] The editing processing unit 21 executes editing of the combined image moving image (deleting a part, splicing, etc.) in response to a user operation.

[0096] The code analysis unit 22 detects a two-dimensional code or a barcode included in the panoramic image and analyzes the two-dimensional code or the barcode to acquire conference participation information. The conference participation information includes information that the device is usable for the conference, the device identification information of the electronic whiteboard 2 stored in the device information storage unit 3001, the conference ID (selected by the user), the IP address of the electronic whiteboard 2, etc. The device identification information may be a serial number or a UUID (Universally Unique Identifier), or may be set by the user. The conference ID is assigned when the conference is reserved or when recording starts. The conference ID may be linked to the conference ID determined by the remote conference service system.

[0097] When the meeting device 60 acquires the conference participation information, the mute request unit 23 requests the interactive whiteboard 2 to mute.

[0098] The voice data analyzer 24 performs spectrum analysis (Fourier transform) on the voice data output by the interactive whiteboard 2 to detect frequencies contained in the voice data, and decodes the conference participation information contained in the voice signal by converting specific frequencies into binary bit data. Note that the voice data analyzer 24 may also obtain the conference participation information by voice recognition.

[0099] The beacon data analysis unit 25 acquires the conference participation information contained in the beacon data transmitted from the meeting device 60 .

[0100] Fig. 10 shows video recording information stored in the information storage unit 1001. The video recording information includes items such as a conference ID, a recording ID, an update date and time, a title, an upload, and a save destination. When a user logs in to the information processing system 50, the information recording application 41 downloads the conference information from the conference information storage unit 5001 of the information processing system 50. The conference ID and other information included in the conference information are reflected in the video recording information. The video recording information in Fig. 10 is held by a terminal device 10 operated by a certain user.

[0101] The conference ID is identification information for identifying the remote conference that has been held. The conference ID is assigned when a remote conference schedule is registered in the conference management system 9, or is assigned by the information processing system 50 in response to a request from the information recording application 41. The conference management system 9 is a system for registering conference and remote conference schedules, a URL (conference link) for starting a remote conference, reservation information for devices to be used in the conference, and the like, and is a scheduler or the like connected from the terminal device 10 via a network. The conference management system 9 can also transmit the registered schedule, etc. to the information processing system 50.

[0102] The recording ID is identification information for identifying the combined image video recorded during a remote conference. The recording ID is assigned by the meeting device 60, but may also be assigned by the information recording application 41 or the information processing system 50. A different recording ID is assigned to the same conference ID when recording stopped during the remote conference but was resumed for some reason.

[0103] The update date and time is the date and time when the combined image video was updated (recording ended). If the combined image video was edited, this is the edit date and time.

[0104] The title is the name of the conference. It may be set when the conference is registered in the conference management system 9, or may be set arbitrarily by the user.

[0105] The "upload" indicates whether the combined image video has been uploaded to the information processing system 50.

[0106] The destination indicates the location (URL or file path) where the combined image video and text data are saved in the storage service system 70. Therefore, the user can view the uploaded combined image video at will. The combined image video and text data are saved with separate file names, for example, following the URL.

[0107] <<Meeting Devices>> Returning to FIG. 9, the explanation will be given. The meeting device 60 has a terminal communication unit 61, a panoramic image creation unit 62, a speaker image creation unit 63, a sound collection unit 64, a voice synthesis unit 65, and a short-range wireless communication unit 66. Each of these units of the meeting device 60 is a function or a means for performing the function, which is realized when any of the components shown in FIG. 5 operates in response to an instruction from the CPU 611 in accordance with a program loaded from the ROM 612 to the DRAM 614. The terminal communication unit 61 communicates with the terminal device 10 using a USB cable or the like. The terminal communication unit 61 may be connected not only by a wired cable, but also by wireless LAN, Bluetooth (registered trademark), or the like.

[0108] The panoramic image creation unit 62 creates a panoramic image. The speaker image creation unit 63 creates speaker images. These creation methods have been explained with reference to Figures 6 and 7. The panoramic image creation unit 62 also serves as an acquisition unit that acquires a two-dimensional code containing conference participation information.

[0109] The sound collection unit 64 converts the sound picked up by the microphone of the meeting device 60 into digital sound data. This allows the sound of what is being said by users and participants at the base of the terminal device 10. The sound collection unit 64 also serves as an acquisition unit that acquires a sound signal containing conference participation information.

[0110] The voice synthesis unit 65 synthesizes the voice transmitted from the terminal device 10 with the voice collected by the voice collection unit 64. As a result, the voices spoken at the other locations 101 and the voices spoken at the own location 102 are combined.

[0111] The short-range wireless communication unit 66 (an example of a first short-range wireless communication unit) receives radio waves (beacons) such as Bluetooth (registered trademark) and Wi-Fi, and extracts data by demodulating the radio waves. The short-range wireless communication unit 66 also serves as an acquisition unit that acquires beacon data that includes conference participation information.

[0112] <<Information Processing System>> The information processing system 50 has a communication unit 51, an authentication unit 52, a screen generation unit 53, a communication management unit 54, a device management unit 55, and a text conversion unit 56. Each of these units of the information processing system 50 is a function or a means for performing a function that is realized when any of the components shown in Fig. 4 operates in response to an instruction from the CPU 501 in accordance with a program loaded from the HD 504 to the RAM 503. The information processing system 50 also has a storage unit 5000 constructed by the HD 504 etc. shown in Fig. 4. The storage unit 5000 has a conference information storage unit 5001, a recording information storage unit 5002, and an association information storage unit 5003 constructed therein.

[0113] The communication unit 51 transmits and receives various information to and from the terminal device 10. For example, the communication unit 51 transmits a list of remote conferences to the terminal device 10 and receives a request to recognize voice data from the terminal device 10.

[0114] The authentication unit 52 authenticates the user operating the terminal device 10. For example, the authentication unit 52 authenticates the user based on whether or not authentication information (user ID and password) included in an authentication request received by the communication unit 51 matches authentication information stored in advance. Note that the authentication information may be an IC card number, biometric authentication information such as face or fingerprint, or the like. The authentication unit 52 may also perform authentication using an external authentication system or an authentication method such as OAUTH.

[0115] The screen generation unit 53 generates screen information to be displayed by the terminal device 10. When the terminal device 10 executes a native application, the screen information is stored in the terminal device 10, and the information to be displayed is transmitted in XML or the like. When the terminal device 10 executes a web application, the screen information is created using HTML, XML, CSS (Cascade Style Sheet), JavaScript (registered trademark), or the like.

[0116] The communication management unit 54 acquires information about the remote conference from the conference management system 9 using each user's account or a system account assigned to the information processing system 50. The communication management unit 54 stores the conference information of the reserved conference in the conference information storage unit in association with the conference ID. The communication management unit 54 also acquires conference information that users belonging to the tenant have permission to view. Since a conference ID is set for each conference, the remote conference and recorded information are associated with each other by the conference ID.

[0117] When the device management unit 55 receives the device identification information of the electronic whiteboard 2 or the meeting device 60, it associates the electronic whiteboard 2 and the meeting device 60 used in the conference with each other for the conference and stores the association information in the association information storage unit 5003. Therefore, the conference ID, the device identification information of the electronic whiteboard 2, and the device identification information of the meeting device 60 are associated with each other. Since the conference ID is also associated with the combined image video, the object data and the combined image video are also associated with each other. Furthermore, when the recording ends (when the conference ends), the device management unit 55 deletes the association from the association information storage unit 5003.

[0118] The text conversion unit 56 converts the voice data requested by the terminal device 10 to be converted into text data into text data using an external voice recognition service system 80. The text conversion unit 56 may also perform the conversion itself.

[0119] FIG. 11 shows an example of conference information stored in the conference information storage unit 5001 managed by the communication management unit 54. The communication management unit 54 can use the above account to obtain a list of remote conferences for which the user belonging to the tenant has viewing authority. In this embodiment, a remote conference is used as an example, but the list of remote conferences also includes conferences held in only one conference room. Viewing authority may be granted to conference information managed by the communication management unit 54 directly from the information recording application of the terminal device 10. Furthermore, the information on remote conferences for which the user belonging to the tenant has viewing authority includes information on conferences created by the user and information on conferences for which the user has been granted viewing authority by other users.

[0120] Conference information is managed by a conference ID and is associated with participants, title (conference name), start date and time, end date and time, location, etc. These are examples of conference information, and conference information may also include other information.

[0121] Participant items are the participants in the meeting.

[0122] The title item indicates the content of the meeting, such as the meeting name and agenda.

[0123] The start date and time item is the date and time when the conference is scheduled to start.

[0124] The end date and time item is the date and time when the conference is scheduled to end.

[0125] The location item is the location where the meeting is held, such as a conference room, a branch office name, or a building.

[0126] The item "electronic whiteboard" is the device identification information of the electronic whiteboard 2 used in the meeting.

[0127] The meeting device item is the device identification information of the meeting device 60 used in the conference.

[0128] As shown in FIGS. 10 and 11, the combined image video recorded at the conference is identified by the conference ID.

[0129] The recording information stored in the recording information storage unit 5002 may be the same as that shown in Fig. 10. However, the information processing system 50 has a list of combined image videos recorded by all users belonging to the tenant. The user may input desired storage destination information (path information such as a URL of a cloud storage system) on a user setting screen of the information recording application 41 of the terminal device 10, and the desired storage destination information may be stored in the recording information storage unit 5002.

[0130] 12 shows correspondence information stored in the correspondence information storage unit 5003, in which the device identification information of the electronic whiteboard 2 and the device identification information of the meeting device 60 are associated with the conference ID. This correspondence information is retained from the time of joining the conference until the end of the conference (leaving the room).

[0131] <<Electronic Whiteboard>> 13 is a functional block diagram explaining the functions of the electronic whiteboard 2 by dividing them into blocks. The electronic whiteboard 2 has a contact position detection unit 31, a drawing data generation unit 32, a data recording unit 33, a display control unit 34, a code generation unit 35, a communication unit 36, an audio data generation unit 37, a short-range wireless communication unit 38, and an I / F 39. Each function of the electronic whiteboard 2 is a function or means realized when any of the components shown in FIG. 8 operates in response to an instruction from the CPU 401 in accordance with a program loaded from the SSD 404 onto the RAM 403.

[0132] The contact position detection unit 31 detects the coordinates of the position where the electronic pen 490 contacts the contact sensor 414. The drawing data generation unit 32 acquires the coordinates where the pen tip of the electronic pen 490 contacts from the contact position detection unit 31. The drawing data generation unit 32 generates stroke data by connecting this sequence of coordinate points by interpolating them.

[0133] The display control unit 34 displays the handwritten data, character strings converted from the handwritten data, and menus for user operation on the display.

[0134] The data recording unit 33 stores handwritten data written on the electronic whiteboard 2, shapes converted from handwritten data such as circles and triangles, stamps such as "Done", PC screens, files, etc. in the object information storage unit 3002. Handwritten data, character strings (including shapes), images such as PC screens, files, etc. are handled as objects. For handwritten data, a group of stroke data becomes one object depending on whether it is a time division due to an interruption in handwriting or a distance division due to a difference in handwriting location.

[0135] The communication unit 36 connects to Wi-Fi or a LAN and communicates with the information processing system 50. The communication unit 36 transmits object information to the information processing system 50, receives object information stored in the information processing system 50 from the information processing system 50, and displays the object information on the display 480.

[0136] The code generation unit 35 encodes the conference participation information into a two-dimensional pattern to generate a two-dimensional code. The code generation unit 35 may encode the conference participation information into a barcode. The code generation unit 35 also serves as an output unit that outputs a two-dimensional code or a barcode containing the conference participation information.

[0137] The audio data generation unit 37 converts the alphabet and numbers into frequencies that last for a fixed period of time, and generates an audio signal by sampling at fixed intervals, similar to PCM conversion. For example, the audio data generation unit 37 represents the alphabet and numbers in ASCII code, with 19 kHz corresponding to binary 0 and 20 kHz corresponding to binary 1. The audio signal is converted to analog by a D / A converter in the speaker 450 and output from the speaker 450. The audio data generation unit 37 also serves as an output unit that outputs an audio signal containing conference participation information.

[0138] The short-range wireless communication unit 38 (an example of a second short-range wireless communication unit) modulates the data, combines it with a carrier wave, and transmits the beacon from the antenna. The short-range wireless communication unit 38 also serves as an output unit that transmits the beacon containing the conference participation information.

[0139] The adjustment unit 39 adjusts the volume of at least one of the microphone 440 and the speaker 450 of the interactive whiteboard 2 in response to a request from the terminal device 10 etc. The adjustment unit 39 mutes the microphone 440 or the speaker 450, for example.

[0140] The electronic whiteboard 2 also has a storage unit 3000 constructed using the SSD 404 and the like shown in Fig. 8. The storage unit 3000 includes a device information storage unit 3001 and an object information storage unit 3002.

[0141] FIG. 14 shows the device identification information and the like stored in the device information storage unit 3001. The device identification information is the device identification information of the electronic whiteboard 2. The IP address is an IP address that allows other devices to connect to the electronic whiteboard 2 via a network. The passcode is used to authenticate other devices when they connect to the electronic whiteboard 2.

[0142] 15 is a diagram illustrating object information stored in the object information storage unit 3002. The object information is information for managing objects displayed on the electronic whiteboard 2. The object information is transmitted to the information processing system 50 and used as minutes of the meeting. If an electronic whiteboard 2 is located at another location when a remote conference is held, the object information is shared.

[0143] The conference ID is identification information of the conference notified by the information processing system 50.

[0144] · Object ID is identification information that identifies an object.

[0145] Type is the type of object, and can be handwritten, text, shape, image, etc. Handwritten text is stroke data (a sequence of coordinate points). Text is a string of characters (character code) converted from handwritten data. Text is sometimes called text data. Shapes are geometric shapes such as triangles and squares converted from handwritten data. Images are image data such as JPEG, PNG, TIFF, etc. imported from a PC or the Internet.

[0146] One screen of the electronic whiteboard 2 is called a page. The item on a page is its page number.

[0147] The coordinates indicate the position of the object relative to a predetermined origin on the electronic whiteboard 2. The object position is, for example, the upper left vertex of the bounding rectangle of the object. The coordinates are expressed, for example, in pixel units of the display.

[0148] The size is the width and height of the object's bounding rectangle.

[0149] <Mute procedure> Below, several muting procedures will be explained, assuming that both the meeting device 60 and the interactive whiteboard 2 are powered on and connected to the network.

[0150] <<Mute Step 1>> FIG. 16 is a sequence diagram showing the flow of processing in which the interactive whiteboard 2 mutes voice input and output in response to participation in a conference.

[0151] S1: First, the user instructs the information recording application 41 running on the terminal device 10 to participate in a conference. The operation reception unit 12 of the information recording application 41 receives the conference participation. Participating in a conference means specifying a conference and transmitting that information to the information processing system 50. By participating in a conference, the device identification information of the meeting device 60 is associated with the conference ID in the association information. In addition, the operation to start recording, which will be described later with reference to FIG. 46, may also serve as conference participation. A request to participate in a conference may start mute control (detection of a two-dimensional code). Note that a conference in a remote conference service system is performed separately from conference participation via mute control. An example of a conference participation screen is shown in FIG. 17.

[0152] S2: The communication unit 11 of the information recording application 41 specifies the conference ID selected by the user and sends a participation request (indicating that the user intends to participate) to the information processing system 50. The communication unit 11 sends the device identification information of the meeting device 60. The communication unit 51 of the information processing system 50 receives the conference participation request (conference ID, device identification information of the meeting device 60). The device management unit 55 associates the device identification information of the meeting device 60 with the conference ID and stores them (the IP address of the terminal device 10 is also obtained).

[0153] This conference ID is a conference ID managed by the conference management system 9, but may also be a conference ID assigned for management purposes by the remote conference service system 90. This associates the conference registered in the remote conference service system 90 with the device used.

[0154] The remote conference application 42 separately communicates with the remote conference service system 90 in response to a user operation, and starts sending and receiving images and audio.

[0155] S3: When participation in the conference is completed, the terminal device 10 starts a process (device detection process) to detect whether there are devices participating in the same conference room. Here, the device detection is performed by capturing images using a camera. The device communication unit 16 requests the meeting device 60 to start capturing images.

[0156] S4: The terminal communication unit 61 of the meeting device 60 receives the image capture start command, and the panoramic image creation unit 62 repeatedly creates panoramic images.

[0157] S5: Next, the user operates the electronic whiteboard 2 to start the conference app and instructs participation in the conference. The contact position detection unit 31 of the electronic whiteboard 2 accepts the conference participation request. The request to participate in the conference may initiate mute control (display of a two-dimensional code). The method of instructing participation in the conference may be the same as that of the terminal device 10.

[0158] S6: The communication unit 36 of the electronic whiteboard 2 specifies the conference ID selected by the user and sends a participation request (indicating that the user intends to participate) to the information processing system 50. The communication unit 36 sends the device identification information of the electronic whiteboard 2. The communication unit 51 of the information processing system 50 receives the conference participation (conference ID, device identification information of the electronic whiteboard 2). The device management unit 55 associates the device identification information of the electronic whiteboard 2 with the conference ID and saves the association information (the IP address of the electronic whiteboard 2 is also known). Because the electronic whiteboard 2 and the meeting device 60 are associated, when the electronic whiteboard 2 sends object data along with the conference ID to the information processing system 50, the object data and the combined image video can be associated and saved as minutes of the same conference.

[0159] S7: When participation in the conference is complete, the code generation unit 35 of the electronic whiteboard 2 generates a two-dimensional code including conference participation information (that the device is usable for the conference, the device identification information of the electronic whiteboard 2, the conference ID, and the IP address of the electronic whiteboard 2). The display control unit 34 displays the two-dimensional code for a certain period of time. An example of the display of the two-dimensional code is shown in FIG. 18. The reason for setting the period of time (for example, several seconds to several minutes) is to prevent the two-dimensional code from interfering with the conference. Note that the conference participation information may include either the device identification information of the electronic whiteboard 2 or the IP address of the electronic whiteboard 2, depending on the communication path used by the terminal device 10 to communicate with the electronic whiteboard 2.

[0160] S8: The panoramic image creation unit 62 of the meeting device 60 generates a panoramic image including the two-dimensional code. The terminal communication unit 61 transmits the panoramic image to the terminal device 10.

[0161] If the two-dimensional code includes information about whether the electronic whiteboard is currently muted, the meeting device 60 may mute its own microphone or speaker. To prevent feedback, the audio input or output of the meeting device 60, or the audio input or output of the electronic whiteboard, may be muted. If the meeting device 60 notices the existence of an electronic whiteboard that is not muted (= has volume) through the two-dimensional code, the meeting device 60 may mute itself.

[0162] S9: When the device communication unit 16 of the terminal device 10 receives a panoramic image, the code analysis unit 22 monitors the panoramic image and constantly determines whether or not it contains a two-dimensional code indicating that the device is to be used in a conference (presence or absence of conference participation information). If no two-dimensional code is contained, the code analysis unit 22 does nothing and continues to wait in a monitoring state until a two-dimensional code is contained.

[0163] If the two-dimensional code indicating that the electronic whiteboard 2 is used in a conference is included, the information recording application 41 detects the presence of the electronic whiteboard 2. The mute request unit 23 compares the conference ID included in the two-dimensional code with the conference ID selected in step S2 to determine whether the electronic whiteboard 2 is participating in the same conference as the meeting device 60. This prevents multiple devices in the same conference room from being associated with different conferences. The determination in step S9 can also be made by the meeting device 60.

[0164] S10: When the meeting device 60 is participating in the same conference as the electronic whiteboard 2, the mute request unit 23 requests the electronic whiteboard 2 to mute at least one of the microphone 440 or the speaker 450 (an example of an adjustment request) via the communication unit 11, with the IP address of the electronic whiteboard 2 as the destination. When the terminal device 10 and the electronic whiteboard 2 are connected to the same LAN (same network address, connected to the same SSID), communication is possible if the IP address of the electronic whiteboard 2 is available. Note that the conference participation information may include an authentication passcode, and in this case, the mute request unit 23 may also transmit the passcode to the electronic whiteboard 2. A preferred example of a method for issuing a mute command is to use a WebAPI (Application Programming Interface) made public by the electronic whiteboard 2, but the method is not limited thereto.

[0165] In step S10, the interactive whiteboard 2 receives a mute request from the terminal device 10, but the interactive whiteboard 2 may also receive a mute request directly from the meeting device 60.

[0166] S11: The communication unit 36 of the interactive whiteboard 2 receives the mute request, and the adjustment unit 39 mutes at least one of the microphone 440 and the speaker 450. Furthermore, since the mute state has been reached, the display control unit 34 ends the display of the two-dimensional code. A screen displayed by the muted interactive whiteboard 2 is shown in FIG. 27.

[0167] If the meeting device 60 and the interactive whiteboard 2 are installed in different conference rooms or are participating in different meetings, for example, and the interactive whiteboard 2 does not receive a mute request, the interactive whiteboard 2 stops displaying the two-dimensional code after a certain time (for example, one minute) has elapsed since displaying the two-dimensional code. In this case, muting is not performed. This prevents the two-dimensional code from remaining displayed. The display time for the two-dimensional code may be set by the user.

[0168] Furthermore, the monitoring process of the two-dimensional code of the terminal device 10 may be terminated after a certain period of time (such as 10 minutes), but since once the process is terminated, it will not be possible to detect devices that have joined the conference after the termination, the monitoring state is continued in this embodiment.

[0169] In addition, in FIG. 16, the user first allows the terminal device 10 to participate in the conference, but the electronic whiteboard 2 may also participate in the conference first or simultaneously.

[0170] 17 is an example of a conference participation screen 300 displayed by the terminal device 10. The conference participation screen 300 is a screen that displays conference information, and has a participation button 301. When the user presses the participation button 301, a conference participation request specifying the conference ID is transmitted to the information processing system 50.

[0171] FIG. 18 shows an example of a two-dimensional code 8 displayed on the interactive whiteboard 2. In FIG. 18, one large two-dimensional code 8 is displayed. The meeting device 60 can easily detect the two-dimensional code 8. A method for displaying the two-dimensional code 8 will be described below.

[0172] FIG. 19 shows an example of the position of the two-dimensional code 8 displayed by the interactive whiteboard 2. As described above, the two-dimensional code 8 includes device identification information and a sign indicating that it is device identification information. On the desk on which the meeting device 60 is placed, there is an obstacle 69 between the meeting device 60 and the interactive whiteboard 2. Note that the meeting device 60 does not need to be a spherical camera to detect the two-dimensional code 8. The interactive whiteboard 2 displays the device identification information above the center line 320 of the display screen of the interactive whiteboard 2 in the up-down direction. This makes it easier for the meeting device 60 to capture the device identification information even if there is an obstacle 69 between the meeting device 60 and the interactive whiteboard 2.

[0173] As shown in FIG. 20, the electronic whiteboard 2 may display the center of the two-dimensional code 8 above the center line 320 of the display screen of the electronic whiteboard 2 in the up-down direction.

[0174] 21, the interactive whiteboard 2 may move the two-dimensional code 8 over time. In FIG. 21, the two-dimensional code 8 moves from left to right. The interactive whiteboard 2 may move the two-dimensional code 8 while it is still displayed, or may display, erase, and then redisplay the code in a different location. This makes it easier for the meeting device 60 to capture the device identification information even if the position of the obstacle 69 is uncertain. Also, even if the interactive whiteboard 2 displays the two-dimensional code 8 in a small size to reduce the sense of oppression felt by the user, the meeting device 60 can easily capture the code.

[0175] Furthermore, the interactive whiteboard 2 may change the size of the two-dimensional code 8 while moving the two-dimensional code 8.

[0176] 22, the interactive whiteboard 2 may simultaneously display multiple two-dimensional codes 8. This makes it easier for the meeting device 60 to capture images of the remaining two-dimensional codes 8 even if some of the two-dimensional codes 8 are hidden by an obstacle 69. Note that the multiple two-dimensional codes 8 may all store the same information, or may store different information.

[0177] 23, the interactive whiteboard 2 may display a two-dimensional code 8 next to (close to) the menu 71. The menu 71 is arranged vertically across the right edge, and is displayed above the center line of the display screen of the interactive whiteboard 2 in the vertical direction, similar to FIG.

[0178] The two-dimensional code 8 is displayed near the menu 71, which makes it less likely for the user to feel uncomfortable. The user can use a wider area of the screen.

[0179] 24, the electronic whiteboard 2 may display a two-dimensional code 8 within a menu 71. Compared to FIG. 23, this makes it easier to eliminate the sense of oppression and allows the user to use a wider screen.

[0180] Fig. 25 is a diagram illustrating a case where the camera of the meeting device 60 is a hemispherical camera. By capturing images of the configurations of Figs. 19 to 24 with a hemispherical camera with a wide horizontal field of view, it becomes easier to find the two-dimensional code 8.

[0181] Note that a barcode instead of the two-dimensional code 8 can also be displayed in the same manner as in Figures 19 to 25. Figure 26 is an example of a barcode 7.

[0182] Note that barcodes 7 are less robust against tilt and other factors than two-dimensional codes 8. For this reason, the code analysis unit 22 cuts out the black and white pattern and corrects the skew angle and pitch angle. The code analysis unit 22 also emphasizes the edges of the black bars. The code analysis unit 22 detects the barcode 7 displayed on the interactive whiteboard 2 by pattern matching the cut-out image with the pattern from the start character to the stop character at the right end, which is registered as the pattern of the barcode 7.

[0183] 19 to 26 illustrate an example in which a two-dimensional code 8 or a barcode 7 is displayed, but the electronic whiteboard 2 may display device identification information (alphabet or numbers), and the information recording application 41 may detect the device identification information by OCR processing.

[0184] Fig. 27 shows an example of a screen 310 displayed on the interactive whiteboard 2 after muting. A mute icon 311 is displayed in Fig. 27. The mute icon 311 allows the user to understand that the interactive whiteboard 2 is currently in a muted state.

[0185] <<Mute Step 2>> 28 is a sequence diagram showing the flow of processing in which the interactive whiteboard 2 mutes audio input and output in response to participation in a conference. In muting procedure 2, the meeting device 60 detects the interactive whiteboard 2 by an audio signal. In the explanation of FIG. 28, only the differences from FIG. 16 will be mainly explained. First, the processing up to participation in the conference, shown in steps S21, S22, S25, and S26, is the same as steps S1, 2, 5, and 6 in FIG. 16.

[0186] S23: When participation in the conference is completed, the terminal device 10 starts a process to detect whether there are other devices participating in the same conference room. Here, the device detection is performed using audio signals. The device communication unit 16 requests the meeting device 60 to start collecting audio.

[0187] S24: The terminal communication unit 61 of the meeting device 60 receives the sound collection start signal, and the sound collection unit 64 repeatedly collects sound.

[0188] S27: When participation in the conference is complete, the audio data generation unit 37 of the interactive whiteboard 2 generates an audio signal including conference participation information and outputs it from the speaker 450. The format of the audio signal is shown in Fig. 29. As shown in Fig. 29, the audio signal includes a preamble indicating the beginning of the audio signal, device identification information of the interactive whiteboard 2, an IP address, and the conference ID of the conference in which the user is participating. Either the device identification information or the IP address is sufficient.

[0189] S28: The sound collection unit 64 of the meeting device 60 generates audio data containing an audio signal. The terminal communication unit 61 transmits the audio data to the terminal device 10.

[0190] S29: When the device communication unit 16 of the terminal device 10 receives the voice data, the voice data analysis unit 24 analyzes the voice data and always determines whether the voice data indicates that the device is used in the conference. Note that the processing of step S29 may be performed in the meeting device 60.

[0191] If the information indicates that the electronic whiteboard 2 is a device used in a conference, the information recording application 41 detects the presence of the electronic whiteboard 2. The mute request unit 23 determines whether the electronic whiteboard 2 is participating in the same conference as the meeting device 60 based on the conference ID included in the audio signal. The subsequent processing may be the same as in FIG. 16 .

[0192] Since detection by sound does not require display on a display or imaging by a camera, it is also possible for the interactive whiteboard 2 to detect the meeting device 60 and mute the meeting device 60 and the terminal device 10. In this case, the roles of the meeting device 60 and the interactive whiteboard 2 are reversed. Also, after the interactive whiteboard 2 detects the meeting device 60, it is also possible for the interactive whiteboard 2 itself to be muted.

[0193] <<Mute step 3>> 30 is a sequence diagram showing the process flow in which the interactive whiteboard 2 mutes audio input and output in response to participation in a conference. In muting procedure 3, the meeting device 60 detects the interactive whiteboard 2 through short-range wireless communication. In the explanation of FIG. 30, only the differences from FIG. 16 will be mainly explained. First, the processes up to participation in the conference, shown in steps S41, S42, S45, and S46, are the same as steps S1, 2, 5, and 6 in FIG. 16.

[0194] S43: When participation in the conference is completed, the terminal device 10 starts a process to detect whether there are other devices participating in the same conference room. Here, the device detection is performed by short-range wireless communication. The device communication unit 16 requests the meeting device 60 to start beacon detection.

[0195] S44: The terminal communication unit 61 of the meeting device 60 receives the beacon detection start signal, and the short-range wireless communication unit 66 repeatedly receives the beacon.

[0196] S47: When participation in the conference is complete, the short-range wireless communication unit 38 of the interactive whiteboard 2 generates a beacon including conference participation information and outputs it from the antenna. The format of the signal included in the beacon may be the same as that shown in FIG. 29. A beacon refers to a wireless radio wave or transmission that repeatedly transmits predetermined information. Any frequency or communication protocol may be used in the beacon. For example, an advertisement packet of Bluetooth Low Energy (registered trademark) is known.

[0197] S48: The short-range wireless communication unit 66 of the meeting device 60 demodulates the beacon to generate beacon data. The terminal communication unit 61 transmits the beacon data to the terminal device 10.

[0198] S49: When the device communication unit 16 of the terminal device 10 receives the beacon data, the beacon data analysis unit 25 analyzes the beacon data and always determines whether the beacon data indicates that the device is to be used in the conference. Note that the processing of step S49 may be performed in the meeting device 60.

[0199] If the information indicates that the electronic whiteboard 2 is a device used in a conference, the information recording application 41 detects the presence of the electronic whiteboard 2. The mute request unit 23 determines whether the electronic whiteboard 2 is participating in the same conference as the meeting device 60 based on the conference ID included in the conference participation information. The subsequent processing may be the same as in FIG. 16 .

[0200] <<Mute step 4>> FIG. 31 is a sequence diagram showing the flow of processing in which the interactive whiteboard 2 mutes audio input and output in response to participation in a conference. Muting procedure 4 begins with an instruction from the user to start device detection. In the explanation of FIG. 31, only the differences from FIG. 16 will be mainly explained. First, the processing up to conference participation, shown in steps S61 to S64, is the same as steps S1, 2, 5, and 6 in FIG. 16. However, both the terminal device 10 and the interactive whiteboard 2 only participate in the conference when they receive a conference participation request, and do not automatically perform mute control (detection and display of two-dimensional codes) when conference participation is completed.

[0201] S65: The user operates the information recording application 41 to instruct the start of device detection processing. The operation reception unit 12 receives the instruction. An example of the device detection start screen is shown in FIG.

[0202] S66: The communication unit 11 of the information recording application 41 specifies the conference ID or the device identification information of the meeting device 60 and transmits a device detection start request to the information processing system 50.

[0203] S67, S68: The communication unit 51 of the information processing system 50 receives the device detection start request. As a result, the device management unit 55 transmits the device detection start request via the communication unit 51 to the meeting devices 60 (terminal devices 10) participating in the conference and the interactive whiteboard 2, which are identified by the conference ID or the device identification information of the meeting devices 60 in the association information. Note that the information processing system 50 may transmit the device detection start request in response to polling from the interactive whiteboard 2, or may transmit the request using two-way communication such as a Web socket.

[0204] As a result, the interactive whiteboard 2 displays the two-dimensional code, and the image is captured by the meeting device 60. The subsequent processing of steps S69 to S75 is the same as steps S3, 4, and 7 to 11 in FIG.

[0205] In the process of FIG. 31, device detection may be performed using an audio signal or a beacon, as in steps 2 and 3 of the mute procedure, instead of a two-dimensional code.

[0206] 32 shows an example of a device detection start screen 330 displayed by the terminal device 10. The device detection start screen 330 has a message 331 saying, "To prevent feedback, detect terminals installed in the same conference room and mute the detected terminals," and a start button 332. The user presses the start button 332 to instruct the start of device detection processing.

[0207] In muting step 4, participating meeting devices 60 and the interactive whiteboard 2 simultaneously start device detection processing in response to a user instruction, making it possible to automatically mute devices through terminal detection even if the timing of each device's joining the meeting differs significantly. For example, even if a user first joins the interactive whiteboard 2 to the meeting and then joins the meeting device 60 a few minutes later, the interactive whiteboard 2 can still be muted. Furthermore, if device detection is not required, device detection will not be performed unnecessarily unless the user issues a start instruction.

[0208] <<Mute step 5>> 33 is a sequence diagram showing the flow of processing in which the interactive whiteboard 2 mutes audio input and output in response to participation in a conference. Muting procedure 5 begins when device detection begins when the interactive whiteboard 2 joins the conference. In the explanation of FIG. 33, only the differences from FIG. 16 will be mainly explained. First, the processing up to conference participation, shown in steps S81 to S84, is the same as steps S1, 2, 5, and 6 in FIG. 16. However, when the terminal device 10 receives the conference participation request, it only participates in the conference, and does not perform mute control (detection of the two-dimensional code) when conference participation is completed.

[0209] S85: The electronic whiteboard 2 receives the instruction to participate in the conference, and when participation in the conference is completed, the display control unit 34 displays a two-dimensional code including the conference participation information.

[0210] S86: Furthermore, when the meeting device 60 and the electronic whiteboard 2 have completed joining the meeting, the device management unit 55 of the information processing system 50 transmits a request to start acquiring the two-dimensional code (a request to start device detection) to the terminal device 10 via the communication unit 51.

[0211] S87: When the communication unit 11 of the information recording application 41 receives the device detection start request, the device communication unit 16 requests the meeting device 60 to start capturing a panoramic image. Because the interactive whiteboard 2 is displaying a two-dimensional code, the terminal device 10 detects the interactive whiteboard 2. The subsequent processing of steps S88 to S92 may be similar to S4, and S8 to S11 in FIG. 16 .

[0212] In FIG. 33, device detection may be performed using an audio signal or a beacon, as in steps 2 and 3 of the mute procedure, instead of a two-dimensional code.

[0213] 33, the meeting device 60 joins the conference first, but the same can be applied to the case where the interactive whiteboard 2 joins the conference first. In this case, the interactive whiteboard 2 displays the two-dimensional code after receiving a notification from the information processing system 50 that the meeting device 60 has joined the conference.

[0214] Furthermore, in muting step 5, the device detection process is executed appropriately each time a second device joins the conference, making it possible to automatically mute by terminal detection without being affected by the timing of the device joining and without requiring any instructions from the user.

[0215] <<Mute step 6>> 34 is a sequence diagram showing the flow of processing in which the interactive whiteboard 2 mutes audio input and output in response to participation in a conference. In muting procedure 6, the terminal device 10 does not directly send a mute instruction to the interactive whiteboard 2, but mutes the interactive whiteboard 2 via the information processing system 50. In the explanation of FIG. 34, only the differences from FIG. 16 will be mainly explained. First, the processing of steps S101 to S109 is the same as S1 to S9 in FIG. 16.

[0216] S110: When the code analysis unit 22 of the information recording application 41 detects the electronic whiteboard 2, the mute request unit 23 requests the information processing system 50 to mute the electronic whiteboard 2 via the communication unit 11. The mute request includes the device identification information of the electronic whiteboard 2 attached thereto so that the information processing system 50 can communicate with the electronic whiteboard 2.

[0217] S111: When the communication unit 51 of the information processing system 50 receives the mute request, it transmits the mute request to the electronic whiteboard 2 identified by the device identification information. For example, after the electronic whiteboard 2 joins a conference, it repeatedly polls specifying the device identification information, and the information processing system 50 transmits the mute request in response to the polling. Alternatively, for example, after the electronic whiteboard 2 joins a conference, the electronic whiteboard 2 and the information processing system 50 communicate via bidirectional communication such as a Web socket (device identification information is transmitted), and the information processing system 50 transmits the mute request to the electronic whiteboard 2 identified by the device identification information.

[0218] S112: When the communication unit 36 of the interactive whiteboard 2 receives the mute request, the adjustment unit 39 mutes at least one of the microphone 440 and the speaker 450.

[0219] In addition, in FIG. 34, device detection may be performed by an audio signal or a beacon, as in steps 2 and 3 of the mute procedure, instead of the two-dimensional code.

[0220] In muting procedure 6, the terminal device 10 mutes via the information processing system 50, so that a mute request can be sent to the electronic whiteboard 2 even in an environment where the terminal device 10 and the electronic whiteboard 2 cannot communicate directly.

[0221] <<Mute step 7>> 35 is a sequence diagram showing the flow of processing in which the interactive whiteboard 2 mutes audio input and output in response to participation in a conference. In muting procedure 7, the remote conference service system 90 stops audio transmission and reception to and from the interactive whiteboard 2, rather than muting at least one of the microphone 440 and speaker 450 of the interactive whiteboard 2. In the explanation of FIG. 35, only the differences from FIG. 16 will be mainly explained. First, the processing of steps S121 to S129 is the same as S1 to S9 in FIG. 16.

[0222] S130: When the code analysis unit 22 of the information recording application 41 detects the electronic whiteboard 2, the mute request unit 23 requests the information processing system 50 to mute the electronic whiteboard 2 via the communication unit 11. The mute request is accompanied by the device identification information or the IP address of the electronic whiteboard 2.

[0223] S131: When the communication unit 51 of the information processing system 50 receives a mute request, the communication unit 51 transmits the mute request to the remote conference service system 90 by specifying the device identification information or IP address of the interactive whiteboard 2.

[0224] S132: The remote conference service system 90 does not transmit audio from other locations to the electronic whiteboard 2 identified by the device identification information or IP address of the electronic whiteboard 2, and does not transmit audio transmitted by the electronic whiteboard 2 to other locations. This results in the same state as when the electronic whiteboard 2 is muted.

[0225] In addition, in FIG. 35, device detection may be performed by an audio signal or a beacon, as in steps 2 and 3 of the mute procedure, instead of the two-dimensional code.

[0226] In step 7 of muting, the remote conference service system 90 mutes the interactive whiteboard 2, so there is no need to change the settings of the device itself.

[0227] <<Mute step 8>> 36 is a sequence diagram showing the flow of processing in which the interactive whiteboard 2 mutes audio input and output in response to participation in a conference. In step 8 of muting, if the interactive whiteboard 2 is already in a muted state when the conference starts, the processing for device detection is not performed. In the explanation of FIG. 36, only the differences from FIG. 16 will be mainly explained. First, the processing of steps S1 to S6 is the same as S1 to S6 in FIG. 16.

[0228] S170: When the communication unit 36 of the interactive whiteboard 2 transmits a request to participate in the conference to the information processing system 50, the adjustment unit 39 determines whether or not it is in a mute state (adjustment state).

[0229] If the adjustment state has already been reached, the display control unit 34 does not display the two-dimensional code, and only when the adjustment state has not been reached, the display control unit 34 displays the two-dimensional code and performs the same process as in muting procedure 1. Note that the terminal device 10 may continue capturing images of the meeting device 60 in order to capture images of the participants in addition to the two-dimensional code. Alternatively, the terminal device 10 may stop capturing images of the meeting device 60 after a certain time has elapsed since joining the meeting.

[0230] It should be noted that device detection may be performed using an audio signal or a beacon, as in steps 2 and 3 of the mute procedure, instead of a two-dimensional code.

[0231] By performing the muting procedure 8, if the interactive whiteboard 2 is already in a muted state when joining the conference, it is possible to prevent unnecessary processing from being executed.

[0232] <Unmute> Next, we will explain some ways to unmute.

[0233] <<Unmute 1>> 37 is a sequence diagram showing the flow of processing in which the electronic whiteboard 2 cancels muting when a participant leaves a conference. When a participant leaves a conference, the object data displayed on the electronic whiteboard 2 is not shared with other locations and is not recorded. In the explanation of FIG. 37, the electronic whiteboard 2 mutes at least one of the microphone 440 and the speaker 450 when the participant joins a conference.

[0234] S141: The user issues an instruction to leave the conference to the interactive whiteboard 2. The instruction is accepted by the operation accepting unit 12. An example of the conference leave instruction screen is shown in FIG.

[0235] S142: In response to the user's instruction, the communication unit 36 of the electronic whiteboard 2 specifies the conference ID and transmits the end of the conference (exit) to the information processing system 50. The communication unit 51 of the information processing system 50 receives the end of the conference, and the device management unit 55 deletes the association information identified by the conference ID.

[0236] S143: In response to the user's instruction to end the conference, the adjustment unit 39 of the interactive whiteboard 2 cancels the mute state and returns the volume to the setting before joining the conference.

[0237] It should be noted that, with regard to the terminal device 10, the user separately performs an operation to end the conference. As a result, the device management unit 55 deletes the device identification information of the meeting device 60 from the correspondence information. With regard to the terminal device 10, no processing related to muting occurs.

[0238] 38 shows an example of a conference exit instruction screen 340 displayed by the electronic whiteboard 2. The conference exit instruction screen 340 has a message 341 asking "Do you want to leave the conference?", an exit button 342, and a cancel button 343. When the user presses the exit button 342, the user instructs the electronic whiteboard 2 to leave the conference.

[0239] Unmuting 1 prevents the electronic whiteboard 2 that was muted when joining the meeting from remaining muted against the user's intention. Note that unmuting 1 can be applied regardless of the method used to mute the board, except for muting step 7.

[0240] <<Unmute 2>> 39 is a sequence diagram showing the flow of processing in which the terminal device 10 unmutes the electronic whiteboard 2 when leaving the conference. In the explanation of FIG. 39, the terminal device 10 mutes the electronic whiteboard 2 when joining the conference.

[0241] S151: The user issues a conference exit instruction to the information recording application 41. The operation reception unit 12 receives the instruction.

[0242] S152: The communication unit 11 of the information recording application 41 sends a mute cancellation request to the interactive whiteboard 2, with the IP address as the destination.

[0243] S153: When the communication unit 36 of the interactive whiteboard 2 receives the mute cancellation request, the adjustment unit 39 cancels the mute state and returns the volume to the setting before joining the conference.

[0244] S154: The communication unit 11 of the information recording application 41 specifies the conference ID and transmits the end of the conference (exit) to the information processing system 50. The communication unit 51 of the information processing system 50 deletes the association information (device identification information of the meeting device 60 and the electronic whiteboard 2) identified by the conference ID.

[0245] In Figure 39, the terminal device 10 sends an unmute instruction directly to the electronic whiteboard 2, but if the terminal device 10 mutes the electronic whiteboard 2 via the information processing system 50, the terminal device 10 may also send an unmute request to the electronic whiteboard 2 via the information processing system 50.

[0246] 39, if the terminal device 10 leaves the conference before the interactive whiteboard 2, it is possible to prevent the interactive whiteboard 2 from remaining muted and preventing the conference from continuing. Also, the user only needs to perform the conference leave operation on the terminal device 10. Note that unmuting 2 can be applied regardless of the method of muting.

[0247] <<Unmute 3>> 40 is a sequence diagram showing the flow of a process for unmuting the electronic whiteboard 2 when communication between the terminal device 10 and the information processing system 50 is disconnected. In the explanation of FIG. 40, the terminal device 10 mutes the electronic whiteboard 2 when participating in a meeting.

[0248] S161, S162: During the conference, the communication unit 51 of the information processing system 50 monitors communication with the terminal device 10 (for example, by sending a Ping command), and if communication is cut off due to a poor network connection or the like, the information processing system 50 detects the cut off of communication with the terminal device 10. Note that the cut off of communication also includes the terminal device 10 shutting down or freezing.

[0249] S163: When the communication disconnection is detected, the device management unit 55 sends an unmute instruction to the electronic whiteboard 2 that has been muted by an instruction from the terminal device 10. The electronic whiteboard 2 that is unmuted is the electronic whiteboard 2 that notified the start of the meeting, and the information processing system 50 holds device identification information, an IP address, and the like.

[0250] S164: When the communication unit 36 of the interactive whiteboard 2 receives the mute cancellation instruction, the adjustment unit 39 cancels the mute state and returns the volume to the setting before joining the conference.

[0251] 40, if communication with the terminal device 10 is cut off, it is possible to prevent the interactive whiteboard 2 from remaining in a muted state, which would prevent the conference from being unable to continue. Note that mute cancellation 1 can be applied regardless of the method of muting.

[0252] <End of device detection on Interactive Board 2> In muting procedure 1 and the like, even if a mute request is not sent to the interactive whiteboard 2, the interactive whiteboard 2 stops displaying the two-dimensional code after a certain time has passed since displaying the two-dimensional code. Therefore, if the interactive whiteboard 2 does not receive a mute request, there is a risk that the two-dimensional code will remain displayed for a certain time. Therefore, the interactive whiteboard 2 stops displaying the two-dimensional code when the user begins to use the interactive whiteboard 2, rather than after the passage of time.

[0253] 41 is a sequence diagram showing the process of ending the display of the two-dimensional code without receiving a mute request after the interactive whiteboard 2 is started up. The processes in steps S181 to S183 may be the same as steps S5 to S7 in FIG.

[0254] S184: The user uses the electronic whiteboard 2. The start of use can be determined when the user touches the screen of the electronic whiteboard 2 with their hand or an electronic pen.

[0255] S185: Alternatively, as a method of determining whether to start using the electronic whiteboard, the terminal device 10 may display a screen on the electronic whiteboard 2. The method of determining whether to start using the electronic whiteboard is not limited to these.

[0256] S186: The terminal device 10 transmits the screen to the interactive whiteboard 2 via an HDMI (registered trademark) cable or the like.

[0257] S187: The display control unit 34 of the electronic whiteboard 2 ends the display of the two-dimensional code when it detects the start of use, even before a certain period of time has elapsed. In this case, the meeting device 60 and the electronic whiteboard 2 are not associated. However, as shown in FIG. 31 , when the information processing system 50 starts device detection as a result of the terminal device 10 joining a meeting, the electronic whiteboard 2 displays the two-dimensional code again, and the meeting device 60 and the electronic whiteboard 2 are associated.

[0258] In addition, in FIG. 41, even if device detection is performed using an audio signal or beacon as in mute steps 2 and 3 instead of a two-dimensional code, device detection can be completed in the same way.

[0259] In the process of FIG. 41, the device detection process ends when the user starts using the interactive whiteboard 2, so it is possible to prevent, for example, a two-dimensional code from interfering with the meeting.

[0260] <Recording during a meeting> Next, several screens that the terminal device 10 displays during a remote conference will be described with reference to Fig. 42 to Fig. 45. Fig. 42 shows an initial screen 200 after login that is displayed by the information recording application 41 running on the terminal device 10. The user of the terminal device 10 connects the information recording application 41 to the information processing system 50. When the user inputs authentication information and successfully logs in, the initial screen 200 of Fig. 42 is displayed.

[0261] The initial screen 200 has a fixed display button 201, a front change button 202, a panoramic image 203, one or more speaker images 204a to 204c (hereinafter, referred to as speaker images 204 when no distinction is made), and a recording start button 205. If the meeting device 60 is already running and capturing images of the surroundings at the time of login, the panoramic image 203 and speaker image 204 created by the meeting device 60 are displayed on the initial screen 200. Therefore, the user can decide whether to start recording while viewing these. If the meeting device 60 is not running (not capturing images), the panoramic image 203 and speaker image 204 are not displayed.

[0262] The information recording application 41 may display speaker images 204 of all participants based on all faces detected from the panoramic image 203, or may display only the speaker images 204 of the N most recently spoken participants. Fig. 42 shows an example in which speaker images 204 of up to three people are displayed. Until a participant speaks, there may be no speaker image 204 (the speaker images are added one by one as a participant speaks), or three speaker images 204 of participants in a predetermined direction may be displayed (the speaker images are replaced as a participant speaks).

[0263] When no one is speaking, such as immediately after the meeting device 60 is started up, a predetermined direction (0°, 120°, 240°, etc.) within the horizontal 360° is created as the speaker image 204. If a fixed display, which will be described later, is set, the fixed display setting takes priority.

[0264] The fixed display button 201 is a button that allows the user to perform an operation to fix a certain area of the panoramic image 203 in close-up as a speaker image 204 .

[0265] The front change button 202 is a button that allows the user to perform an operation to change the front of the panoramic image 203 (because a panoramic image covers 360° horizontally, the right and left edges are aligned). The user can slide the panoramic image 203 left and right with a pointing device to determine which participant will be shown in the front. The user's operation is transmitted to the meeting device 60, which then changes the angle of the front within the 360° horizontal direction, creates a panoramic image, and transmits it to the terminal device 10.

[0266] When the user presses the record start button 205, the information recording application 41 displays the recording setting screen 210 shown in FIG.

[0267] 43 is an example of a recording setting screen 210 displayed by the information recording application 41. On the recording setting screen 210, the user can set whether to record (whether to include in the recording) the panoramic image and speaker image created by the meeting device 60, and the desktop screen or the screen of a running application of the terminal device 10. If the information recording application 41 does not record either the panoramic image and speaker image, or the desktop screen or the screen of a running application, only the audio (audio output by the terminal device 10 + audio collected by the meeting device 60) is recorded.

[0268] The camera toggle button 211 is a button for switching on and off the recording of the panoramic image and the speaker image created by the meeting device 60. The camera toggle button 211 may be set to record the panoramic image and the speaker image separately.

[0269] The PC screen toggle button 212 is a button that switches on and off recording of the desktop screen of the terminal device 10 and the screen of an application running on the terminal device 10. When the PC screen toggle button 212 is in the on state, the desktop screen is recorded.

[0270] If the user wants to record the screen of an application, the user further selects the application in the application selection field 213. The application selection field 213 displays the names of applications currently running on the terminal device 10 in a pull-down menu. The user can select the application to record. The information recording application 41 acquires this application name from the OS. The information recording application 41 can display only those applications that have a UI (screen) among the currently running applications. The selected applications may include the remote conference application 42. Therefore, the information recording application 41 can record videos of materials displayed in the remote conference application 42 and participants at each location. In addition, the applications displayed in the pull-down menu include various applications currently running on the terminal device, such as a presentation application, a word processing application, a spreadsheet application, an application for creating and editing documents or other materials, a cloud electronic whiteboard application, and a web browser application. Therefore, the user can flexibly select the application screens to include in the combined image video.

[0271] When recording on an app basis, the user can select multiple apps, and the information recording app 41 can record the screens of all the selected apps.

[0272] When both the camera toggle button 211 and the PC screen toggle button 212 are off, the recording content confirmation window 214 displays "Only audio will be recorded." This audio includes the audio output by the terminal device 10 (audio received by the remote conference app 42 from other locations 101) and the audio collected by the meeting device 60. In other words, as long as a remote conference is being held, the audio from the remote conference app 42 and the audio from the meeting device 60 are saved regardless of whether images are recorded. However, the user may be able to selectively stop saving the audio from the remote conference app 42 and the audio from the meeting device 60 using user settings.

[0273] The combined image moving image is recorded as follows according to the combination of on / off of the camera toggle button 211 and the PC screen toggle button 212. The combined image moving image is also displayed in real time in the recording content confirmation window 214.

[0274] When the camera toggle button 211 is on and the PC screen toggle button 212 is off, the panoramic image captured by the meeting device 60 and the speaker image are displayed in the recording content confirmation window 214 .

[0275] When the camera toggle button 211 is off and the PC screen toggle button 212 is on (the screen has also been selected), the recording content confirmation window 214 displays the desktop screen or the screen of the selected application.

[0276] When the camera toggle button 211 is on and the PC screen toggle button 212 is on, the panoramic image and speaker image captured by the meeting device 60, as well as the desktop screen and the screen of the selected application, are displayed side by side in the recording content confirmation window 214.

[0277] Therefore, there are cases where the panoramic image, speaker image, and app screen are not recorded, or where the panoramic image, speaker image, and app screen are not recorded at all, but in this embodiment, for convenience, the image created by the information recording application 41 is referred to as a combined image video.

[0278] Furthermore, the recording setting screen 210 has a check box 215 along with the message "Automatically transcribe recording after uploading." The recording setting screen 210 also has a Start Recording Now button 216. When the user checks the check box 215, text data obtained by converting comments made during the remote conference is attached to the recorded video. In this case, after recording ends, the information recording application 41 uploads the audio to the information processing system 50 along with a request to convert it into text data. Furthermore, when the user presses the Start Recording Now button 216, the recording in progress screen 220 shown in FIG. 44 is displayed.

[0279] FIG. 44 is an example of a recording screen 220 that the information recording application 41 displays during recording. Note that the explanation of FIG. 44 will mainly focus on the differences from FIG. 42. The recording screen 220 displays in real time a combined image video being recorded under the conditions set by the user on the recording setting screen 210. The recording screen 220 of FIG. 44 is displayed when the camera toggle button 211 is on and the PC screen toggle button 212 is off, and displays a panoramic image 203 and a speaker image 204 (both of which are videos) created by the meeting device 60. The recording screen 220 displays a recording icon 225, a pause button 226, and an end recording button 227.

[0280] The pause button 226 is a button for stopping recording, and after stopping, recording can also be resumed. The end recording button 227 is a button for ending recording. The pause button 226 does not change the recording ID, but the end recording button 227 changes the recording ID. After pausing, when recording is resumed, the user can reset the recording conditions set on the recording setting screen 210. In this case, the information recording application 41 may create multiple recording files each time recording is stopped (for example, by pressing the end recording button 227), or may combine multiple files so that they form a single continuous video (for example, by pressing the pause button 226). Furthermore, when the information recording application 41 plays back a combined image video, it may play back multiple recording files continuously as a single video.

[0281] Furthermore, the recording in progress screen 220 has an Get Information From Calendar button 221, a conference name field 222, a time field 223, and a location field 224. The Get Information From Calendar button 221 is a button that the user uses to obtain conference information from the conference management system 9. When the Get Information From Calendar button 221 is pressed, the information recording application 41 obtains and displays a list of conferences that the user has permission to view from the information processing system 50. The user selects the remote conference to be held from the conference list. This causes the conference information to be reflected in the conference name field 222, time field 223, and location field 224. The conference name field 222 reflects the title of the conference information, the time field 223 reflects the start time and end time, and the location field 224 reflects the location. Furthermore, the conference information and recording information in the conference management system are associated with each other by a conference ID.

[0282] Once the remote conference is over and the user stops recording, a combined image video with audio is created.

[0283] FIG. 45 is an example of the conference list screen 230 displayed by the information recording application 41. The conference list screen 230 is a list of conferences, and can display a list of recorded information recorded during remote conferences. It also includes conferences that were held only in a certain conference room, regardless of whether they were remote conferences. The conference list screen 230 displays conference information that the logged-in user has permission to view in the conference information storage unit 5001. Information on video recording information stored in the information storage unit 1001 may be further integrated.

[0284] The conference list screen 230 is displayed when the user selects the conference list tab 231 on the initial screen 200 in FIG. 42. The conference list screen 230 displays a list 236 of recorded information that the user has permission to view. The conference creator (minutes creator) can set viewing permissions for participants. The conference list may be a list of stored recorded information or a list of conference schedules and conference data.

[0285] The conference list screen 230 has the following items: a check box 232 , an update date and time 233 , a title 234 , and a status 235 .

[0286] The checkboxes 232 are used to select recorded files. The checkboxes 232 are used when the user wants to delete recorded files all at once.

[0287] The update date and time 233 indicates the start and end times of recording the combined image video. If the video has been edited, the edit date and time may be used.

[0288] The title 234 is the title of the meeting (agenda, etc.). This may be transcribed from the meeting information or set by the user.

[0289] The status 235 indicates whether the combined image video has been uploaded to the information processing system 50. If it has not been uploaded, "Local PC" is displayed, and if it has been uploaded, "Uploaded" is displayed. If it has not been uploaded, an upload button is displayed. If there is a combined image video that has not been uploaded, it is recommended that the information recording application 41 automatically uploads it when the user logs in to the information processing system 50.

[0290] When the user selects an arbitrary title or the like from the list 236 of combined image videos with a pointing device, the information recording application 41 displays a recording playback screen, but this is not shown in this embodiment. On the recording playback screen, it is possible to play back the combined image video, etc.

[0291] It is desirable that users be able to narrow down conferences by update date and time, title, keywords, etc. Also, if there are a large number of conferences displayed and it is difficult to find the relevant conference, it is desirable that a search function be provided that allows users to input words and phrases to narrow down recorded information based on words contained in the remarks made at the conference or the title, etc. The search function allows users to find the desired recorded information in a short time even when there is a large amount of recorded information. Furthermore, on the conference list screen, users may be able to sort conferences by update date and time or title.

[0292] <<Recording operation or processing>> 46 is an example of a sequence diagram showing a procedure for recording a panoramic image, a speaker image, and an application screen by the information recording application 41. Conference participation and mute control have already been completed.

[0293] S201: The user operates the teleconferencing app 42 to start a teleconferencing. Here, it is assumed that the teleconferencing app 42 of the user's own base 102 and the teleconferencing app 42 of another base 101 have started a teleconferencing. The teleconferencing app 42 of the user's own base 102 transmits images captured by a camera and audio collected by a microphone of the meeting device 60 to the teleconferencing app 42 of the other base 101. The teleconferencing app 42 of the other base 101 displays the received images on a display and outputs the received audio from a speaker. Similarly, the teleconferencing app 42 of the other base 101 transmits images captured by a camera and audio collected by a microphone of the meeting device 60 to the teleconferencing app 42 of the user's own base 102. The teleconferencing app 42 of the user's own base 102 displays the received images on a display and outputs the received audio from a speaker. Each teleconferencing app 42 repeats this process to realize the teleconferencing.

[0294] S202: The user makes settings related to recording on the recording setting screen 210 of the information recording application 41 shown in Fig. 43. The operation accepting unit 12 of the information recording application 41 accepts the settings. Here, it is assumed that the camera toggle button 211 and the PC screen toggle button 212 are both on.

[0295] S203: When the user operates to start recording, the recording control unit 17 of the information recording application 41 starts recording.

[0296] S204: The application screen acquisition unit 14 of the information recording application 41 requests the application screen selected by the user (more specifically, the application screen acquisition unit 14 acquires the application screen via the OS). In FIG. 46, the application selected by the user is the remote conference application 42.

[0297] S205: The recording control unit 17 of the information recording application 41 notifies the meeting device 60 of the start of recording via the device communication unit 16. When notifying, the recording control unit 17 may also notify that the camera toggle button 211 is on (requesting a panoramic image and a speaker image). Regardless of whether a request is made, the meeting device 60 sends the panoramic image and the speaker image to the information recording application 41.

[0298] S206: When the terminal communication unit 61 of the meeting device 60 receives the start of recording, it assigns a unique recording ID and returns the recording ID to the information recording application 41. The recording ID may be assigned by the information recording application 41 or may be obtained from the information processing system 50.

[0299] S207: The voice acquisition unit 15 of the information recording application 41 acquires the voice data output by the terminal device 10 (voice data received by the remote conference application 42).

[0300] S208: The device communication unit 16 transmits the voice data acquired by the voice acquisition unit 15 and a synthesis request to the meeting device 60.

[0301] S209: The terminal communication unit 61 of the meeting device 60 receives the voice data and the synthesis request, and the voice synthesis unit 65 synthesizes the received voice data with the surrounding voice data collected by the sound collection unit 64. For example, the voice synthesis unit 65 adds the two pieces of voice data together. Because clear voices from around the meeting device 60 are recorded, the accuracy of converting voices from around the meeting device 60 (conference room side) into text is improved.

[0302] This voice synthesis can also be performed by the terminal device 10. The recording function may be distributed to the meeting device 60, and the voice processing may be distributed to the terminal device 10. In this case, the load on the meeting device 60 can be reduced.

[0303] S210: Furthermore, the panoramic image creating unit 62 of the meeting device 60 creates a panoramic image, and the speaker image creating unit 63 creates a speaker image.

[0304] S211: The device communication unit 16 of the information recording application 41 repeatedly acquires the panoramic image and the speaker image from the meeting device 60. The device communication unit 16 also repeatedly acquires the synthesized audio data from the meeting device 60. These acquisitions may be performed by the device communication unit 16 making a request to the meeting device 60. Alternatively, the meeting device 60 may automatically send the panoramic image and the speaker image upon receiving a notification that the camera toggle button 211 is on. The meeting device 60 may automatically send the synthesized audio data to the information recording application 41 upon receiving a request to synthesize audio data.

[0305] S212: The recording control unit 17 of the information recording application 41 creates a combined image by arranging the application screen, panoramic image, and speaker image acquired from the remote conference application 42 side by side. The recording control unit 17 repeatedly creates combined images and creates a combined image video by specifying each combined image as a frame that constitutes the video. The recording control unit 17 also saves the audio data received from the meeting device 60.

[0306] The information recording application 41 repeats the above steps S207 to S212.

[0307] S213: When the remote conference is over and there is no longer any need to record, the user instructs the information recording application 41 to end recording (for example, by pressing the recording end button 227). The operation reception unit 12 of the information recording application 41 receives the instruction.

[0308] S214: The device communication unit 16 of the information recording application 41 notifies the meeting device 60 that recording has ended. The meeting device 60 continues creating panoramic images and speaker images and synthesizing audio. However, the meeting device 60 may change the processing load, such as changing the resolution or fps, depending on whether recording is in progress.

[0309] S215: The recording control unit 17 of the information recording application 41 combines the audio data with the combined image moving image to create a combined image moving image with audio.

[0310] S216: Furthermore, if the user checks the checkbox 215 associated with "Automatically transcribe recording after uploading" on the recording setting screen 210, the audio data processing unit 18 requests the information processing system 50 to convert the audio data into text data. In detail, the audio data processing unit 18 specifies the URL of the save destination via the communication unit 11, and transmits a request to the information processing system 50 to convert the audio data combined into the combined image video together with the conference ID and recording ID.

[0311] S217: The communication unit 51 of the information processing system 50 receives the request to convert the voice data, and the text conversion unit 56 converts the voice data into text data using the voice recognition service system 80. The communication unit 51 saves the text data in the same storage destination (the URL of the storage service system 70) as the storage destination of the combined image video. In the recorded video information storage unit 5002, the text data is associated with the combined image video by the conference ID and the recorded video ID. The text data may be managed by the communication management unit 54 of the information processing system 50 and stored in the storage unit 5000. Alternatively, the terminal device 10 may request voice recognition from the voice recognition service system 80 and store the text data acquired from the voice recognition service system 80 in a storage destination. The voice recognition service system 80 returns the converted text data to the information processing system 50, or may directly send it to the URL of the storage destination. The voice recognition service system 80 may select or switch between multiple services depending on setting information set by the user in the information processing system 50.

[0312] S218: The upload unit 20 of the information recording application 41 also saves the combined image moving image in the destination for saving the combined image moving image via the communication unit 11. The combined image moving image is associated with the conference ID and the recording ID in the recording information storage unit 5002. The combined image moving image is recorded as having been uploaded.

[0313] S219: The user inputs the end of the meeting into the electronic whiteboard 2. The user may input the end of the meeting into the terminal device 10, and the end of the meeting may be transmitted from the terminal device 10 to the electronic whiteboard 2. In this case, the end of the meeting may be transmitted to the electronic whiteboard 2 via the information processing system 50.

[0314] S220: The communication unit 36 of the electronic whiteboard 2 specifies the conference ID and transmits object data displayed (for example, handwritten) during the conference to the information processing system 50. The communication unit 36 may also transmit device identification information of the electronic whiteboard 2 to the information processing system 50. In this case, the conference ID is identified by the association information.

[0315] S221: The information processing system 50 stores the object data in the same storage location as the combined image video and the like based on the conference ID.

[0316] The save destination is notified to the user, so the user can share the combined image video with participants by informing them of the save destination by email, etc. Even if the combined image video, audio data, text data, and object data are created using different devices, they can be collected and stored in a single storage location, making them easy for users to view later.

[0317] The processing of steps S207 to S212 does not have to be performed in the order shown in FIG. 46, and the synthesis of the audio data and the creation of the combined image may be performed in the reverse order.

[0318] <Major Effects> In this way, the meeting device 60 of this embodiment mutes the interactive whiteboard 2 displaying the two-dimensional code, so the correct device can be muted without mistaking the device. Furthermore, a user may want to output audio from the meeting device 60 located at the center of the meeting participants and mute the microphone 440 that collects the audio from the interactive whiteboard 2, but such control was not possible with conventional technology. As described above, this embodiment prevents the device from being misplaced.

[0319] <Other application examples> The best mode for carrying out the present invention has been described above using examples, but the present invention is not limited to these examples in any way, and various modifications and substitutions can be made within the scope that does not deviate from the gist of the present invention.

[0320] For example, the terminal device 10 and the meeting device 60 may be integrated. The meeting device 60 may be externally attached to the terminal device 10. The meeting device 60 may also be an omnidirectional camera, a microphone, and a speaker connected by a cable.

[0321] Furthermore, a meeting device 60 may also be placed at another location 101. The other location 101 separately uses the meeting device 60 to create combined image video and text data. Furthermore, multiple meeting devices 60 may be placed at one location. In this case, multiple pieces of record information are created for each meeting device 60.

[0322] The layout of the panoramic image 203, the speaker image 204, and the app screen in the combined image video used in this embodiment is merely an example. The panoramic image 203 may be on the bottom and the speaker image 204 on top, or the user may change the layout, or may individually switch between displaying and hiding the panoramic image 203 and the speaker image 204 during playback.

[0323] Furthermore, the configuration examples in Figure 9 and the like are divided according to main functions to facilitate understanding of the processing by the terminal device 10, the meeting device 60, and the information processing system 50. The method of dividing the processing units and the names thereof do not limit the present invention. The processing by the terminal device 10, the meeting device 60, and the information processing system 50 can be divided into even more processing units depending on the processing content. Furthermore, the processing can also be divided so that one processing unit includes even more processes.

[0324] Additionally, the devices described in the examples are merely illustrative of one of several computing environments for implementing the embodiments disclosed herein. In one embodiment, information processing system 50 includes multiple computing devices, such as a server cluster, configured to communicate with each other via any type of communication link, including a network, shared memory, etc., and to perform the processes disclosed herein.

[0325] Furthermore, the information processing system 50 can be configured to share the disclosed processing steps, such as those shown in FIG. 16, in various combinations. For example, a process executed by a specific unit can be executed by multiple information processing devices included in the information processing system 50. Furthermore, the information processing system 50 may be integrated into a single server device, or may be divided into multiple devices.

[0326] Each function of the above-described embodiments can be realized by one or more processing circuits. Here, the term "processing circuit" in this specification includes a processor programmed to perform each function by software, such as a processor implemented by an electronic circuit, as well as devices such as an ASIC (Application Specific Integrated Circuit), a DSP (Digital Signal Processor), an FPGA (Field Programmable Gate Array), and a conventional circuit module designed to perform each of the above-described functions. [Explanation of symbols]

[0327] 10 Terminal Equipment 50 Information Processing Systems 60 Meeting Devices 100 Recording Information Creation System [Prior art documents] [Patent documents]

[0328] [Patent Document 1] JP 2014-143534 A

Claims

1. a first device having an audio input / output means; a second device having an audio input means and an audio output means, The first device is an acquisition unit that acquires information about the second device output by the second device; a terminal communication unit that transmits the information about the second device acquired by the acquisition unit to a terminal device, The second device is an output unit that outputs information about the second device; an adjustment unit that adjusts the volume of at least one of the audio input means and the audio output means of the second device in response to a request for adjusting the volume of the audio received from the terminal device via a network for the second device; An equipment system having:

2. the output unit displays information about the second device; The equipment system according to claim 1 , wherein the acquisition unit acquires the information about the second equipment displayed by the output unit by capturing an image with an imaging unit.

3. the output unit outputs information about the second device by voice; The device system according to claim 1 , wherein the acquisition unit acquires the information about the second device output by the output unit in the form of voice by collecting the voice with a microphone.

4. the output unit transmits information about the second device by short-range wireless communication; The equipment system according to claim 1 , wherein the acquisition unit acquires the information about the second equipment by receiving the information about the second equipment transmitted by short-range wireless communication.

5. an information processing system capable of communicating with the first device and the second device; The second device is When a request to participate in the communication is received, a notification of participation is sent to the information processing system; the output unit starts outputting information about the second device, the first device is communicably connected to the terminal device, When the terminal device receives a request to participate in the communication, the terminal device transmits a notice of participation to the information processing system; Start acquiring information about the second device; 5. The equipment system according to claim 1, wherein the information processing system associates the first equipment with the second equipment.

6. an information processing system capable of communicating with the first device and the second device; The second device is When a request to participate in the communication is received, a notification of participation is sent to the information processing system; the output unit starts outputting information about the second device, the first device is communicably connected to the terminal device, When the terminal device receives a request to participate in the communication, the terminal device transmits a notice of participation to the information processing system; The information processing system includes: An equipment system according to any one of claims 1 to 4, characterized in that when a request to participate in the communication is received from the terminal device and the second device, the equipment system requests the first device to start acquiring information about the second device via the terminal device.

7. the information about the second device is an IP address of the second device; The equipment system described in any one of claims 1 to 6, characterized in that the second equipment receives a request to adjust the volume of the audio from the terminal device connected to the same network, addressed to the IP address.

8. the information about the second device is device identification information of the second device, The equipment system according to claim 5, characterized in that when the information processing system receives a request to adjust the volume of the audio from the terminal device by specifying the device identification information of the second device, the information processing system transmits the request to adjust the volume of the audio to the second device identified by the device identification information of the second device.

9. The equipment system according to claim 5, characterized in that when the information processing system receives a request from the terminal device to adjust the volume of the audio by specifying the device identification information of the second device, it requests the remote conference service system to stop transmitting audio to the second device identified by the device identification information of the second device and to stop transmitting audio transmitted from the second device to other locations.

10. The equipment system according to claim 5, characterized in that, when the second device receives a request to join a communication, if the volume of the audio is already adjusted, the second device does not start outputting information about the second device.

11. The device system according to any one of claims 1 to 8, wherein the second device cancels the volume adjustment when the second device accepts the end of communication.

12. An equipment system according to any one of claims 1 to 8, wherein when the terminal device accepts the end of communication, the second device receives a request to cancel the volume adjustment from the terminal device, and when the second device receives the request to cancel the volume adjustment from the terminal device, it cancels the volume adjustment.

13. When the information processing system detects a disconnection of communication with the terminal device, it transmits a request to cancel the volume adjustment to the second device; 9. The device system according to claim 5, wherein the second device cancels the volume adjustment when the second device receives a request to cancel the volume adjustment.

14. the second device outputs information about the second device only for a certain period of time after displaying the information about the second device, 9. The equipment system according to claim 1, wherein when an operation on the second equipment is detected, output of information on the second equipment is terminated even within a certain period of time.

15. a first device having an audio input / output means; a second device having an audio input means and an audio output means, The first device is an acquisition unit that acquires information about the second device output by the second device; a terminal communication unit that transmits the information about the second device acquired by the acquisition unit to a network, The second device is an output unit that outputs information about the second device; an adjustment unit that adjusts the volume of at least one of the audio input means and the audio output means of the second device in response to a request for adjusting the volume of the audio received via a network for the second device; An equipment system having:

16. a first device having an audio input / output means; a second device having an audio input means and an audio output means; and a volume adjustment method performed by an equipment system having the second device, The first device is an acquisition unit acquiring information about the second device output by the second device; a step of transmitting the information about the second device acquired by the acquisition unit to a terminal device by a terminal communication unit; The second device is an output unit outputting information about the second device; an adjustment unit adjusting the volume of at least one of the audio input means or the audio output means of the second device in response to a request for adjusting the volume of the audio received from the terminal device via a network for the second device; A volume adjustment method having the above.

17. an acquisition unit that acquires information about a second device output by the second device; a second device capable of communicating with a first device via a network, the second device having a terminal communication unit that transmits information about the second device acquired by the acquisition unit to a terminal device, and an audio input / output means, audio input means and audio output means; an output unit that outputs information about the second device; an adjustment unit that adjusts the volume of at least one of the audio input means and the audio output means of the second device in response to a request for adjusting the volume of the audio received from the terminal device via a network for the second device; A second device having:

18. audio input means and audio output means; an output unit that outputs information about the second device; a first device capable of communicating with a second device via a network, the first device having an adjustment unit that adjusts a volume of at least one of an input means or an output means of the audio of the second device in response to a request for adjusting the volume of the audio of the second device received from a terminal device via a network, an audio input / output means; an acquisition unit that acquires information about the second device output by the second device; a terminal communication unit that transmits the information about the second device acquired by the acquisition unit to a terminal device;

19. audio input means and audio output means; an output unit that outputs information about the second device; an adjustment unit that adjusts a volume of at least one of an input means or an output means of the audio of the second device in response to a request for adjusting the volume of the audio of the second device received from a terminal device via a network, said first device being capable of communicating with a second device, audio input means and output means; an acquisition unit that acquires information about the second device output by the second device; The first device adjusts the volume of at least one of the audio input means and the audio output means in accordance with the information about the second device acquired by the acquisition unit.

Images