Conference device, equipment system, echo suppression method, and program

Abstract

To suppress echoes.SOLUTION: A conference device 60 according to the present invention having a microphone 608, a speaker 619, and a noise reduction function includes an echo signal output unit that outputs an echo signal from a speaker, an echo component detection unit that detects an echo component from the echo signal output by the speaker and input to the microphone, and a control unit that controls the level of noise reduction by the noise reduction function in accordance with the magnitude of the echo component amount detected by the echo component detection unit.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to a conference device, an equipment system, an echo suppression method, and a program. [Background technology]

[0002] There is a known remote communication system in which images and audio are transmitted in real time from one location to one or more other locations, and users in remote locations hold a conference using images and audio. In remote communication, a conference device equipped with a microphone and a speaker may be used.

[0003] The microphones and speakers used in a conference are equipped with an echo canceller that reduces echo (see, for example, Patent Document 1). Patent Document 1 discloses a technology for suppressing echo by adjusting the gain of the microphone according to the calculated value of an adaptive filter. Summary of the Invention [Problem to be solved by the invention]

[0004] However, the conventional technology has a problem that the speaker volume is easily reduced by the echo canceller. The amount of echo components can be reduced by reducing the microphone volume and speaker volume. However, since the speaker volume is changed every time the adaptive filter is calculated, the volume of the voice that the user hears is constantly fluctuating, making it difficult to hear.

[0005] In view of the above problems, an object of the present invention is to suppress echoes. [Means for solving the problem]

[0006] In view of the above problems, the present invention provides a conference device having a microphone, a speaker, and a noise reduction function, comprising: an echo signal output unit that outputs an echo signal from the speaker; an echo component detection unit that detects an echo component from the echo signal output by the speaker and input to the microphone; and a control unit that controls the level of noise reduction by the noise reduction function in accordance with the magnitude of the echo component detected by the echo component detection unit. Effect of the Invention

[0007] The present invention is capable of suppressing echoes. [Brief description of the drawings]

[0008] [Figure 1] 11A and 11B are diagrams for explaining an outline of the creation of record information that saves the screen of an application executed during a remote conference together with a panoramic image of the surroundings. [Diagram 2] FIG. 1 is a diagram for explaining an outline of a process for reducing echoes in a record information creation system. [Diagram 3] FIG. 1 illustrates an example of the configuration of a record information creation system. [Figure 4] FIG. 2 is a diagram illustrating an example of a hardware configuration of an information processing system and a terminal device. [Diagram 5] FIG. 2 is a diagram illustrating an example of a hardware configuration of a meeting device. [Figure 6] FIG. 2 is a diagram illustrating an imaging range of a meeting device. [Figure 7] 1A and 1B are diagrams illustrating the cutting out of a panoramic image and a speaker image. [Figure 8] FIG. 2 is a diagram illustrating a hardware configuration of an example of an electronic whiteboard. [Figure 9] 1 is an example of a functional block diagram illustrating the functions of a terminal device, a meeting device, and an information processing system in a record information creation system, divided into blocks. [Figure 10] 4 is a diagram showing an example of moving image recording information stored in an information storage unit; FIG. [Figure 11]FIG. 4 is an example of a functional block diagram illustrating in detail the function of a sound collection unit. [Figure 12] FIG. 4 is a diagram illustrating an example of conference information managed by a communication management unit. [Figure 13] 11 is a diagram showing an example of association information stored in an association information storage unit, in which a conference ID is associated with device identification information; FIG. [Figure 14] FIG. 2 is an example of a functional block diagram illustrating functions of an electronic whiteboard divided into blocks. [Figure 15] 4 is a diagram showing an example of device identification information etc. stored in a device information storage unit; FIG. [Figure 16] 4 is a diagram illustrating object information stored in an object information storage unit. FIG. [Figure 17] FIG. 1 is a conceptual diagram of a microphone and speaker mounted on a meeting device. [Figure 18] FIG. 2 is a diagram illustrating the principle of echo generation. [Figure 19] FIG. 1 is a diagram for explaining two echo cancellation methods (an echo suppressor method and an adaptive echo canceller method). [Figure 20] 2 is a diagram illustrating an example of a functional configuration and audio processing performed by an audio processing unit; FIG. [Figure 21] 1 is a sequence diagram illustrating an example of a flow of voice data transmission from another location to a home location in a remote conference. [Figure 22] 4A and 4B are diagrams illustrating the magnitude of a speaker output signal and a microphone input signal. [Figure 23] 11 is a flowchart illustrating a process in which a meeting device at a local base receives a pseudo echo signal from a meeting device at another location and controls a noise reduction level; FIG. [Figure 24] 11 is a diagram showing an example of a relationship between a noise reduction level and a magnitude of an echo component amount in several speech sections. [Diagram 25] FIG. 13 is a diagram showing an example of an initial screen displayed by an information recording application running on the terminal device after login. [Figure 26] FIG. 13 is a diagram showing an example of a recording setting screen displayed by an information recording application. [Figure 27] FIG. 13 is a diagram showing an example of a recording screen displayed by the information recording application during recording. [Figure 28] FIG. 13 is a diagram showing an example of a conference list screen displayed by the information recording application. [Figure 29] 11 is a sequence diagram illustrating an example of a procedure in which an information recording application records a panoramic image, a speaker image, and an application screen. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] An equipment system and an echo cancellation method performed by the equipment system will be described below as an example of an embodiment of the present invention.

[0010] <An example of how to create minutes for a remote meeting> First, an outline of a method for creating minutes using a panoramic image and an application screen will be described with reference to Fig. 1. Fig. 1 is a diagram for explaining an outline of creating record information that saves the screen of an application executed during a remote conference together with a panoramic image of the surroundings. As shown in Fig. 1, a user at a home base 102 shown in the figure is holding a remote conference with another base 101 using a remote conference service system 90.

[0011] The record information creation system 100 of this embodiment creates record information (such as minutes) using a horizontal panoramic image (hereinafter referred to as a panoramic image) acquired by processing information captured by a meeting device 60 equipped with an imaging means capable of capturing 360° images of the surroundings, a microphone, and a speaker, and a screen created by an application (hereinafter referred to as an app) executed by a terminal device 10. Regarding audio, the record information creation system 100 synthesizes the audio received by a remote conference application 42 and the audio acquired by the meeting device 60, and includes the synthesized audio in the record information. An outline will be given below.

[0012] (1) An information recording application 41 and a remote conference application 42, which will be described later, are running on the terminal device 10. In addition, an application for displaying materials may also be running. The information recording application 41 transmits audio output by the terminal device 10 (including audio received by the remote conference application from other locations) to the meeting device 60. The meeting device 60 mixes (combines) the audio it has acquired with the audio of the remote conference application 42.

[0013] (2) The meeting device 60 is equipped with a microphone, and performs processing to extract the speaker from the panoramic image based on the direction from which the voice was picked up, to create a speaker image. The meeting device 60 transmits both the panoramic image and the speaker image to the terminal device 10.

[0014] (3) The information recording application 41 running on the terminal device 10 can display the panoramic image 203 and the speaker image 204. The information recording application 41 combines any application screen selected by the user (for example, the screen 103 of the remote conference application 42) with the panoramic image 203 and the speaker image 204. For example, the panoramic image 203, the speaker image 204, and the application screen 103 are combined (hereinafter referred to as a combined image 105) so that the panoramic image 203 and the speaker image 204 are arranged on the left side and the screen 103 of the remote conference application 42 is arranged on the right side. The application screen is an example of screen information (described later) displayed by each application such as the remote conference application. Since the process of (3) is executed repeatedly, the combined image 105 becomes a moving image (hereinafter referred to as a combined image moving image). The information recording application 41 also combines the synthesized audio with the combined image moving image to create a moving image with audio.

[0015] In this embodiment, an example in which the panoramic image 203, the speaker image 204, and the application screen 103 are combined will be described, but the information recording application 41 may store these separately and arrange them on the screen during playback.

[0016] (4) The information recording application 41 accepts editing operations (cutting out unnecessary parts by the user) and completes the combined image movie. The combined image movie constitutes a part of the recorded information.

[0017] (5) The information recording application 41 transmits the created combined image video (with audio) to the storage service system 70 and stores it there.

[0018] (6) The information recording application 41 also extracts only the audio from the combined image video (the audio before combination may be kept) and transmits the extracted audio to the information processing system 50. The information processing system 50 transmits the audio to the voice recognition service system 80, which converts the audio into text data, and converts the audio into text. The text data also includes data on how many minutes after the start of recording that the speech occurred.

[0019] In the case of real-time text conversion, the meeting device 60 transmits the voice directly to the information processing system 50. The meeting device 60 transmits the text data to the information recording application 41 in real time.

[0020] (7) The information processing system 50 additionally stores the text data in the storage service system 70 that stores the combined image video. The text data constitutes a part of the recorded information.

[0021] The information processing system 50 can perform billing processing for the user according to the service used. For example, the billing is calculated based on the amount of text data, the file size of the combined image video, the processing time, and the like.

[0022] In this way, the combined image video displays a panoramic image of the surroundings including the user and an image of the speaker, and further displays the screen of an application displayed during the remote conference, such as the remote conference application 42. When a participant or a non-participant in the remote conference views the combined image video as minutes, the situation during the remote conference is reproduced with a sense of realism.

[0023] <Outline of processing or operation> Fig. 2 is a diagram for explaining an outline of the process of reducing echo in the record information creation system 100. Note that a system in which the meeting device 60 at the other site (an example of a first conferencing device) and the meeting device 60 at the own site (an example of a second conferencing device) are taken out of the record information creation system 100 as shown in Fig. 2 is called an equipment system.

[0024] (1) The meeting device 60 at the other site outputs a pseudo echo signal from the speaker 619. The pseudo echo signal is made as inaudible as possible to the human ear, for example a mosquito sound, so as not to hinder conversation. The output pseudo echo signal is input to the microphone 608 of the meeting device 60 at the other site.

[0025] (2) The meeting device 60 at the other location transmits a pseudo echo signal to the terminal device 10 at the other location via the terminal device 10 and the remote conference service system 90 .

[0026] (3) The terminal device 10 at the own base transmits the received pseudo echo signal to the meeting device 60 at the own base. The meeting device 60 outputs the pseudo echo signal from the speaker 619, so that the pseudo echo signal is input to the microphone 608.

[0027] (4) When the meeting device 60 at the own site detects the pseudo echo signal, it detects the echo component based on the sound pressure of the pseudo echo signal. The meeting device 60 at the own site controls the noise reduction level according to the magnitude of the echo component.

[0028] In this way, the meeting device 60 of this embodiment can reduce echo by adjusting the noise reduction level according to the amount of echo components. Since the configuration suppresses echo by the noise reduction level rather than the gain, it is possible to provide a sound that is easy to hear without fluctuations in the gain (volume) and frequency characteristics (tone) of the sound heard by the user. In addition, since an echo signal that is difficult for humans to hear is transmitted, it is possible to measure the amount of echo components and suppress the echo without disrupting the conversation.

[0029] <Terminology> Echo is a phenomenon in which your voice is output from the speaker of the other party and then input to the microphone of the other party, and is returned to your own speaker. Echo is known to cause feedback.

[0030] Echo cancellation means reducing echoes, and this function is called an echo canceller. Echo cancellation can also reduce feedback.

[0031] The echo signal or pseudo echo signal is a sound output from the speaker of the meeting device 60 in order to detect the magnitude of the echo component. The echo signal is preferably a sound in a frequency band that is inaudible to humans so as not to disturb conversation.

[0032] The echo component is the sound that returns to the speaker on your side due to echo. When an echo canceller is used, it refers to the sound data that cannot be completely canceled by the echo canceller.

[0033] Noise reduction (NR) is a function that analyzes audio data and reduces or removes only the noise components that constantly occur, such as ambient noise, making the audio data easier to recognize.

[0034] The noise reduction level is the degree to which noise is reduced. The higher the noise reduction level, the more noise is reduced.

[0035] The conference device is a device that has a microphone, a speaker, and a noise reduction function. The conference device is a device that is mainly used in a conference. The conference device is described as a meeting device 60 in this embodiment.

[0036] The image information of the surroundings of the meeting device acquired by the meeting device is image information acquired by the meeting device by imaging the surrounding space (for example, a space of 180° to 360° in the horizontal direction) surrounding the meeting device, and refers to an image acquired by performing a predetermined process on the image information of the curved surface imaged by the meeting device. The predetermined process is various processes for creating the surrounding image information from the captured information, such as flattening the image of the curved surface. The predetermined process may include a process for creating the surrounding image, a process for cutting out the speaker image, and a process for combining the surrounding image and the speaker image. In this embodiment, the surrounding image is described by the term panoramic image. A panoramic image is an image with a field angle of approximately 180° to 360° in the horizontal direction. It is not necessary for one meeting device to capture the panoramic image, and a plurality of imaging devices with normal fields of view may be combined. The meeting device is intended to be installed and used on a table or other suitable location for meetings at a base or for grasping the surrounding situation, but it may also be used for surveillance (security, disaster prevention, etc.), monitoring (childcare, elderly care, etc.), and on-site situation analysis (solutions, marketing, etc.).

[0037] The recorded information is information recorded by the information recording application 41, and is stored and preserved in a viewable manner as information linked to the identification information of a certain conference (meeting), and includes, for example, the following information:

[0038] Video information created based on the screen information displayed by the selected app (e.g., a remote conferencing app) and image information of the device's surroundings acquired by the device.

[0039] During a meeting, audio information is acquired and synthesized between the remote conference app (terminal device) and the meeting device at the base.

[0040] -Text information converted from the captured voice.

[0041] Other data and images related to the meeting, such as document files used during the meeting, notes added, translations of text data, images and stroke data created using the cloud electronic whiteboard service during the meeting, etc.

[0042] etc. When the information recording application 41 records the screen of the remote conference application or the state of the conference at the base, the recorded information may be the minutes of the conference that was held. The minutes are an example of recorded information, and the recorded information may be called differently depending on the remote conference or the contents of the conference at the base, and may be called, for example, a record of communication or a record of the base situation. The recorded information includes files in a number of formats, such as video files (combined image video, etc.), audio files, text data (text data in which audio is recognized), document files, image files, and table format files, and since the files are associated with each other with the identification information of the conference, they can be viewed chronologically at the time of viewing, either together or selectively.

[0043] A tenant is a group of users (such as a company, a local government, or some of these organizations) who have made a contract to receive a service from a service provider. In this embodiment, the creation of record information and conversion to text data are performed because the tenant has a contract with the service provider.

[0044] Remote communication refers to communication through voice and video with a person at a physically distant location by utilizing software and terminal devices. One example of remote communication is a remote conference, which may also be called a meeting, a meeting, a consultation, an application for a contract, a gathering, a get-together, a seminar, a course, a study group, a seminar, a training session, etc.

[0045] A base is a place where activities are based. An example of a base is a conference room. A conference room is a room that is set up mainly for use in meetings. A base can also be a home, a reception desk, a store, a warehouse, an outdoor site, or any other location or space where terminal equipment, devices, etc. can be installed.

[0046] Although voice refers to speech sounds uttered by humans, surrounding sounds, and the like, and voice data refers to voice that has been digitized, in the present embodiment, a strict distinction will not be made in the description.

[0047] <System configuration example> Next, the system configuration of the record information creation system 100 will be described with reference to Fig. 3. Fig. 3 shows an example of the configuration of the record information creation system 100. Fig. 3 shows one base (own base 102) among multiple bases where a remote conference is held, and a terminal device 10 at the own base 102 communicates with an information processing system 50, a storage service system 70, and a remote conference service system 90 via a network. A meeting device 60 and an electronic whiteboard 2 are also arranged at the own base 102, and the terminal device 10 is connected to the meeting device 60 so as to be able to communicate with each other via a USB cable or the like.

[0048] At least an information recording application 41 and a remote conference application 42 run on the terminal device 10. The remote conference application 42 can communicate with the terminal devices 10 at other locations 101 via a remote conference service system 90 on the network, allowing users at each location to hold a conference from a remote location. The information recording application 41 creates record information for the remote conference held by the remote conference application 42, using the functions of the information processing system 50 and the meeting device 60.

[0049] In this embodiment, an example of creating record information during a remote conference will be described, but the conference does not have to be a conference that communicates with a remote location. In other words, the conference may be a conference in which only participants at one location participate. In this case, the images captured by the meeting device 60 and the collected audio are stored without being synthesized, and there is no change in the processing of the information recording application 41.

[0050] The terminal device 10 has a built-in camera with a normal angle of view (or it may be external), which captures an image of the front including the user 107 who operates the terminal device 10. The normal angle of view is an image that is not a panoramic image, and in this embodiment, it is a planar image that is not curved such as a spherical image. Therefore, the user can hold a conventional remote conference using the remote conference application 42 without being aware of the information recording application 41. The information recording application 41 and the meeting device 60 do not affect the remote conference application 42 except for an increase in the processing load of the terminal device 10. The remote conference application 42 can also transmit the panoramic image and the speaker image captured by the meeting device 60 to the remote conference service system 90.

[0051] The information recording application 41 creates recorded information by communicating with the meeting device 60. The information recording application 41 also performs operations such as synthesizing the audio acquired by the meeting device 60 with the audio received by the remote conference application 42 from another location. The meeting device 60 is a meeting device equipped with a panoramic image capturing device, a microphone, and a speaker. The camera of the terminal device 10 can only capture an image of a limited range in front, but the meeting device 60 can capture an image of the entire surroundings (not necessarily the entire surroundings) surrounding the meeting device 60. The meeting device 60 can always capture multiple participants 106 shown in FIG. 3 within its angle of view.

[0052] In addition, the meeting device 60 cuts out a speaker image from a panoramic image. The meeting device 60 is not limited to being placed on a desk, but may be placed anywhere in the own base 102. Since the meeting device 60 can capture a celestial sphere image, it may be placed on the ceiling, for example.

[0053] The information recording application 41 displays a list of applications running on the terminal device 10, synthesizes images for the recorded information (creates a combined image video), plays back the combined image video, accepts editing, etc. The information recording application 41 also displays a list of remote conferences that have been held or are scheduled to be held, etc. The list of remote conferences is used for information related to the recorded information, and allows the user to link the remote conferences with the recorded information.

[0054] The remote conference application 42 is an application for a terminal device to perform remote communication with other terminal devices by establishing a communication connection with other terminal devices at the other base 101, transmitting and receiving images and audio, displaying images and outputting audio, etc. The remote conference application can also be called a remote communication application, a remote information common application, etc.

[0055] The information recording application 41 and the remote conference application 42 may be Web applications or native applications. A Web application is an application that performs processing by cooperation between a program on a Web server and a program on a Web browser or a native application, and does not need to be installed on the terminal device 10. A native application is an application that is installed on the terminal device 10 and used. In this embodiment, both applications will be described as native applications.

[0056] The terminal device 10 may be a general-purpose information processing device equipped with a communication function, such as a PC (Personal Computer), a smartphone, a tablet terminal, etc. The terminal device 10 may also be an electronic whiteboard 2, a game machine, a PDA (Personal Digital Assistant), a wearable PC, a car navigation system, an industrial machine, a medical device, a network home appliance, etc. The terminal device 10 may be any device on which at least the information recording application 41 and the remote conference application 42 can run.

[0057] The electronic whiteboard 2 displays data handwritten on a touch panel by an input means such as a pen or a finger on a display. The electronic whiteboard 2 can communicate with a terminal device 10 or the like by wire or wirelessly, and can import a screen displayed by the terminal device 10 and display it on the display. The electronic whiteboard 2 can convert handwritten data into text data and share information displayed on the display with electronic whiteboards 2 at other locations. The electronic whiteboard 2 may be a simple whiteboard that does not have a touch panel and projects an image using a projector. The electronic whiteboard 2 may also be a tablet terminal, a notebook PC (Personal Computer), a PDA (Personal Digital Assistant), a game console, or the like equipped with a touch panel.

[0058] The electronic whiteboard 2 can communicate with the information processing system 50. After the electronic whiteboard 2 is powered on, for example, the electronic whiteboard 2 can receive information from the information processing system 50 by polling the information processing system 50 or by using two-way communication such as a Web socket.

[0059] The information processing system 50 is one or more information processing devices arranged on a network. The information processing system 50 has one or more server applications that perform processing in cooperation with the information recording application 41, and a platform service. The server applications manage remote conference lists, manage recorded information recorded in remote conferences, and manage various settings and storage paths. The platform service performs user authentication, contracts, billing, and the like.

[0060] All or part of the functions of the information processing system 50 may exist in a cloud environment or an on-premise environment. The information processing system 50 may be configured with multiple server devices or may be configured with one information processing device. For example, the server application and the platform service may be provided by different information processing devices, and further, an information processing device may exist for each function in the server application. The information processing system 50 may be integrated with the storage service system 70 and the voice recognition service system 80 described below.

[0061] The storage service system 70 is a storage means on a network, and provides a storage service that accepts the storage of files, etc. Known examples of the storage service system 70 include One Drive (registered trademark), Google Workspace (registered trademark), DropBox (registered trademark), etc. The storage service system 70 may be an on-premise NAS (Network Attached Storage), etc.

[0062] The voice recognition service system 80 provides a service of performing voice recognition on voice data and converting it into text data. The voice recognition service system 80 may be a general-purpose commercial service, or may be part of the functions of the information processing system 50. In addition, the voice recognition service system 80 may be set to use a different service system for each user, tenant, or conference.

[0063] <Hardware configuration example> The hardware configuration of the information processing system 50 and the terminal device 10 according to this embodiment will be described with reference to FIG.

[0064] <<Information processing system and terminal device>> Fig. 4 is a diagram showing an example of a hardware configuration of an information processing system 50 and a terminal device 10 according to this embodiment. As shown in Fig. 4, the information processing system 50 and the terminal device 10 are constructed by a computer, and include a CPU 501, a ROM 502, a RAM 503, a HD (Hard Disk) 504, a HDD (Hard Disk Drive) controller 505, a display 506, an external device connection I / F (Interface) 508, a network I / F 509, a bus line 510, a keyboard 511, a pointing device 512, an optical drive 514, and a media I / F 516.

[0065] Among these, the CPU 501 controls the overall operation of the information processing system 50 and the terminal device 10. The ROM 502 stores programs such as an IPL used to drive the CPU 501. The RAM 503 is used as a work area for the CPU 501. The HD 504 stores various data such as programs. The HDD controller 505 controls the reading or writing of various data from the HD 504 under the control of the CPU 501. The display 506 displays various information such as a cursor, a menu, a window, a character, or an image. The external device connection I / F 508 is an interface for connecting various external devices. In this case, the external device is, for example, a USB (Universal Serial Bus) memory or a printer. The network I / F 509 is an interface for data communication using a network. The bus line 510 is an address bus, a data bus, or the like for electrically connecting each component such as the CPU 501 shown in FIG. 4.

[0066] Moreover, the keyboard 511 is a type of input means having a plurality of keys used for inputting characters, numbers, various instructions, etc. The pointing device 512 is a type of input means for selecting and executing various instructions, selecting a processing target, moving a cursor, etc. The optical drive 514 controls reading and writing of various data from an optical storage medium 513, which is an example of a removable storage medium. The optical storage medium 513 may be a CD, a DVD, Blu-ray (registered trademark), etc. The media I / F 516 controls reading and writing (storing) of data from a storage medium 515 such as a flash memory.

[0067] <<Meeting Devices>> The hardware configuration of the meeting device 60 will be described with reference to FIG. 5. FIG. 5 is an example of a hardware configuration diagram of the meeting device 60 capable of capturing 360° video. In the following, the meeting device 60 is assumed to be a device that uses an imaging element to capture 360° video around the device at a predetermined height, but the imaging element may be one or any number of imaging elements, two or more. In addition, it does not necessarily have to be a dedicated device, and a PC, digital camera, smartphone, etc. may be provided with substantially the same function by attaching a 360° video imaging unit to be attached later.

[0068] As shown in FIG. 5, the meeting device 60 has an imaging unit 601, an image processing unit 604, an imaging control unit 605, a microphone 608, a sound processing unit 609, a CPU (Central Processing Unit) 611, a ROM (Read Only Memory) 612, an SRAM (Static Random Access Memory) 613, a DRAM (Dynamic Random Access Memory) 614, an operation unit 615, an external device connection I / F 616, a communication unit 617, an antenna 617a, an audio sensor 618, and a speaker 619.

[0069] Among these, the imaging unit 601 includes a wide-angle lens (so-called fisheye lens) 602 having a 360° angle of view for forming a hemispherical image, and an imaging element 603 (image sensor) provided corresponding to each wide-angle lens. The imaging element 603 includes an image sensor such as a CMOS (Complementary Metal Oxide Semiconductor) sensor or a CCD (Charge Coupled Device) sensor that converts an optical image by the fisheye lens 602 into image data of an electrical signal and outputs it, a timing generation circuit that generates a horizontal or vertical synchronization signal or a pixel clock for this image sensor, and a group of registers in which various commands and parameters necessary for the operation of this imaging element are set. The imaging unit 601 may be a 360° camera, which is an example of an imaging means capable of capturing an image of 360° around the meeting device 60.

[0070] The imaging elements 603 (image sensors) of the imaging unit 601 are connected to the image processing unit 604 via a parallel I / F bus. Meanwhile, the imaging elements 603 of the imaging unit 601 are connected to the imaging control unit 605 via a serial I / F bus (such as an I2C bus). The image processing unit 604, the imaging control unit 605, and the sound processing unit 609 are connected to a CPU 611 via a bus 610. Furthermore, a ROM 612, an SRAM 613, a DRAM 614, an operation unit 615, an external device connection I / F 616, a communication unit 617, an audio sensor 618, a speaker 619, and the like are also connected to the bus 610.

[0071] The image processing unit 604 takes in the image data output from the image sensor 603 via the parallel I / F bus, performs a predetermined process on the image data, and creates data on a panoramic image and a speaker image from the fisheye video. Furthermore, the image processing unit 604 synthesizes the panoramic image and the speaker image, etc., and outputs one video.

[0072] The imaging control unit 605 generally uses an I2C bus to set commands and the like in the registers of the imaging element 603, with the imaging control unit 605 acting as a master device and the imaging element 603 acting as a slave device. Necessary commands and the like are received from the CPU 611. The imaging control unit 605 also uses the I2C bus to retrieve status data and the like from the registers of the imaging element 603 and send it to the CPU 611.

[0073] Furthermore, the imaging control unit 605 instructs the imaging element 603 to output image data when the imaging start button on the operation unit 615 is pressed or when an imaging start instruction is received from a PC. Some meeting devices 60 have a preview display function or a function corresponding to video display on a display (for example, a PC or smartphone display). In this case, the image data is output from the imaging element 603 continuously at a predetermined frame rate (frames / second).

[0074] As described later, the imaging control unit 605 also functions as a synchronization control means that cooperates with the CPU 611 to synchronize the output timing of image data from the imaging element 603. In this embodiment, the meeting device 60 is not provided with a display, but a display unit may be provided.

[0075] Microphones 608a, 608b, and 608c (hereinafter referred to as microphones 608 when no distinction is made) convert sound into sound (signal) data. Sound processing unit 609 takes in the audio data output from microphones 608a, 608b, and 608c via an I / F bus, mixes the audio data, and performs predetermined processing. Sound processing unit 609 also determines the direction of the sound source (speaker) from the sound level (volume) input from microphones 608a, 608b, and 608c. Speaker 619 converts the input audio data into sound.

[0076] The meeting device 60 detects a silent section when the signal levels from the microphones 608a, 608b, and 608c to the sound processing unit 609 and the signal level from the speaker 619 fall below a predetermined value. Immediately after detection, the meeting device 60 transmits a pseudo echo signal that has been created in advance by the sound processing unit 609. In this embodiment, the speaker 619 is configured to output the pseudo echo signal, so the speaker 619 needs to output a high frequency of up to 16,000 Hz. On the other hand, the speaker 619 and the pseudo echo transmitter may be separate entities.

[0077] The speaker 619 is often a box-shaped speaker with multiple diaphragms, and if it is a separate unit, it is only necessary to output low and medium frequencies below 8000 Hz, which is a cost-effective configuration, but the installation space for the pseudo echo transmitter becomes an issue. When the pseudo echo signal output by the speaker 619 returns to the home base via another base, the signal is input from the terminal device 10 to the sound processing unit 609, but the microphone 608 needs to collect high frequencies up to 16000 Hz. In this embodiment, the microphone 608 and the pseudo echo signal receiver are integrated together, just like the speaker.

[0078] The CPU 611 controls the overall operation of the meeting device 60 and executes necessary processing. The ROM 612 stores various programs for operating the meeting device 60. The SRAM 613 and DRAM 614 are work memories, and store programs executed by the CPU 611, data in the middle of processing, etc. In particular, the DRAM 614 stores image data in the middle of processing by the image processing unit 604, and data of a processed equirectangular projection image.

[0079] The operation unit 615 is a general term for operation buttons such as an imaging start button 615a. By operating the operation unit 615, the user can start imaging or recording, as well as turn the power on / off, establish communication connections, and input settings such as various imaging modes and imaging conditions.

[0080] The external device connection I / F 616 is an interface for connecting various external devices. In this case, the external device is, for example, a PC (Personal Computer), a display, a projector, an electronic whiteboard, etc. The external device connection I / F 616 may include, for example, a USB terminal, an HDMI (registered trademark) terminal, etc. The video data and image data stored in the DRAM 614 are transmitted to an external terminal via the external device connection I / F 616, or recorded in an external medium. In addition, the meeting device 60 may use the multiple external device connection I / Fs 616 to, for example, transmit image information captured and acquired by the meeting device 60 to a PC via USB and record it, while acquiring video from the PC to the meeting device 60 (for example, screen information displayed by a remote conference application, etc.), and further transmit the same from the meeting device 60 to other external devices (display, projector, electronic whiteboard, etc.) via HDMI (registered trademark) for display.

[0081] The communication unit 617 may communicate with a cloud server via the Internet using a wireless communication technology such as Wi-Fi via an antenna 617a provided in the meeting device 60, and transmit the stored video data and image data to the cloud server. The communication unit 617 may also be capable of communicating with nearby devices using a short-range wireless communication technology such as BLE (Bluetooth Low Energy, registered trademark) or NFC.

[0082] The audio sensor 618 is a sensor that acquires 360° audio information in order to identify from which direction the loudest audio is input in 360° around (horizontal plane) the meeting device 60. The sound processing unit 609 identifies the strongest direction based on the input 360° audio parameters and outputs the audio input direction in 360°.

[0083] Other sensors (such as a direction / acceleration sensor or GPS) may calculate direction, position, angle, acceleration, etc., and use them for image correction and addition of location information.

[0084] The image processing unit 604 also creates a panoramic image in the following way: The CPU 611 performs predetermined camera image processing such as Bayer conversion (RGB complementation processing) on ​​the RAW data input from the image sensor that inputs the spherical image to create a fisheye image (curved image). It then performs DeWarp processing (distortion correction processing) on ​​the created fisheye image (curved image) to create a panoramic image (flat image) that shows 360° around the meeting device 60.

[0085] The CPU 611 creates speaker images in the following way. The CPU 611 creates speaker images by cutting out the speaker from a panoramic image (flat image) showing 360° of the surroundings. The CPU 611 cuts out the speaker image from the panoramic image, taking the voice input direction identified from 360° output using the audio sensor 618 and the sound processing unit 609 as the speaker's direction. The method of cutting out a person's image from the voice input direction at this time is to cut out a 30° area centered on the identified voice direction from 360°, and perform face detection within that area to cut out the image. The CPU 611 further identifies speaker images for a specific number of people (e.g., three people) who have spoken most recently from the cut out speaker images.

[0086] The panoramic image and the one or more speaker images may be sent separately to the information recording application 41, or the meeting device 60 may create one image from them and send it to the information recording application 41. In this embodiment, the panoramic image and the one or more speaker images are sent separately from the meeting device 60 to the information recording application 41.

[0087] Fig. 6 is a diagram explaining the imaging range of the meeting device 60. As shown in Fig. 6(a), the meeting device 60 captures an image in a range of 360° in the horizontal direction. As shown in Fig. 6(b), the imaging range of the meeting device 60 is 0° in the horizontal direction at the height of the meeting device 60, and a certain angle above and below.

[0088] FIG. 7 is a diagram for explaining the cutting out of a panoramic image and a speaker image. As shown in FIG. 7, the image captured by the meeting device 60 forms a part 110 of a sphere, and therefore has a three-dimensional shape. As shown in FIG. 6(b), the meeting device 60 performs perspective projection transformation by dividing the angle of view into predetermined angles in the vertical direction and predetermined angles in the horizontal direction. By performing perspective projection transformation over the entire horizontal direction of 360° without gaps, a predetermined number of planar images are obtained, and a panoramic image 111 is obtained by connecting the predetermined number of planar images to the left and right. In addition, the meeting device 60 performs face detection from the panoramic image in a predetermined range centered on the voice direction, and creates a speaker image 112 by cutting out 15° left and right from the center of the face (30° in total).

[0089] <<Electronic whiteboard>> Fig. 8 is a hardware configuration diagram of the electronic whiteboard 2. As shown in Fig. 8, the electronic whiteboard 2 includes a CPU (Central Processing Unit) 401, a ROM (Read Only Memory) 402, a RAM (Random Access Memory) 403, an SSD (Solid State Drive) 404, a network I / F 405, and an external device connection I / F (Interface) 406.

[0090] Of these, the CPU 401 controls the overall operation of the electronic whiteboard 2. The ROM 402 stores programs used to start the OS, such as an IPL (Initial Program Loader). The RAM 403 is used as a work area for the CPU 401. The SSD 404 stores various data, such as programs for the electronic whiteboard 2. The network I / F 405 controls communication with a communication network. The external device connection I / F 406 is an interface for connecting various external devices. In this case, the external devices are, for example, a USB (Universal Serial Bus) memory 430 and external devices (a microphone 440 (an example of an input means), a speaker 450 (an example of an output means), and a camera 460).

[0091] The electronic whiteboard 2 also includes a capture device 411, a GPU 412, a display controller 413, a contact sensor 414, a sensor controller 415, an electronic pen controller 416, a short-range communication circuit 419, an antenna 419a of the short-range communication circuit 419, a power switch 422, and selection switches 423.

[0092] Among these, the capture device 411 displays display information on the display of an external PC (Personal Computer) 470 as a still image or a moving image. The GPU (Graphics Processing Unit) 412 is a semiconductor chip that specializes in graphics. The display controller 413 controls and manages the screen display to output an output image from the GPU 412 to the display 480 or the like. The contact sensor 414 detects that the electronic pen 490 or the user's hand H or the like has contacted the display 480. The sensor controller 415 controls the processing of the contact sensor 414. The contact sensor 414 inputs and detects coordinates by an infrared blocking method. The method of inputting and detecting the coordinates is a method in which two light receiving and emitting devices installed at both ends of the upper side of the display 480 emit a plurality of infrared rays in parallel to the display 480, and receive the light that is reflected by a reflecting member installed around the display 480 and returns along the same optical path as the optical path of the light emitted by the light receiving element. The light receiving and emitting device, which is the contact sensor 414, outputs the position (position on the light receiving element) where the infrared ray is blocked by the object to the sensor controller 415, and the sensor controller 415 specifies the coordinate position, which is the contact position of the object, from these two pieces of position information. The electronic pen controller 416 determines whether the pen tip or the pen tail touches the display 480 by communicating with the electronic pen 490 via Bluetooth (registered trademark). The short-range communication circuit 419 is a communication circuit such as NFC (Near Field Communication) or Bluetooth (registered trademark). The power switch 422 is a switch for switching ON / OFF the power of the electronic whiteboard 2. The selection switches 423 are, for example, a group of switches for adjusting the brightness and shade, color, etc. of the display 480.

[0093] Furthermore, the electronic whiteboard 2 includes a bus line 410. The bus line 410 is an address bus, a data bus, or the like for electrically connecting the components such as the CPU 401 shown in FIG.

[0094] The contact sensor 414 is not limited to an infrared blocking type, and may be a capacitive touch panel that identifies the contact position by detecting a change in capacitance. The contact sensor 414 may be a resistive film type touch panel that identifies the contact position by a voltage change in two opposing resistive films. The contact sensor 414 may be an electromagnetic induction type touch panel that identifies the contact position by detecting electromagnetic induction caused by a contact object touching the display unit. In addition, the contact sensor 414 may use various detection means. The electronic pen controller 416 may determine whether or not a touch has been made on not only the tip and end of the electronic pen 490, but also on a part of the electronic pen 490 that is held by the user or on other parts of the electronic pen.

[0095] <About the function> Next, the functional configuration of the record information creation system 100 will be described with reference to Fig. 9. Fig. 9 is an example of a functional block diagram illustrating the functions of the terminal device 10, the meeting device 60, and the information processing system 50 in the record information creation system 100 by dividing them into blocks.

[0096] <<Terminal Device>> The information recording application 41 that runs on the terminal device 10 includes a communication unit 11, an operation reception unit 12, a display control unit 13, an application screen acquisition unit 14, an audio acquisition unit 15, a device communication unit 16, a recording control unit 17, an audio data processing unit 18, a recording and playback unit 19, an upload unit 20, an edit processing unit 21, and a code analysis unit 22. Each of these units in the terminal device 10 is a function or a means that is realized by operating any of the components shown in Fig. 4 according to an instruction from the CPU 501 in accordance with the information recording application 41 that is expanded from the HD 504 to the RAM 503. The terminal device 10 also includes a storage unit 1000 that is constructed by the HD 504 and the like shown in Fig. 4. An information storage unit 1001 is constructed in the storage unit 1000.

[0097] The communication unit 11 communicates various information with the information processing system 50 via the network. The operation reception unit 12 receives various operations for the information recording application 41. The display control unit 13 displays various screens that serve as a user interface in the information recording application 41 according to screen transitions set in the information recording application 41.

[0098] The application screen acquisition unit 14 acquires screen information displayed by an application selected by the user, or screen information of a desktop screen, etc., from an OS (Operating System) or the like. When the application selected by the user is a remote conference application 42, a screen generated by the remote conference application 42 (an image captured by the terminal device camera of each site, a display image of the shared material, and an image including participant icons and participant names, etc.) is acquired. The screen information displayed by the application (application screen) is information displayed as a window by a running application and acquired as an image by an information recording application. The window of the application is drawn as an area of ​​the window as an area in the entire desktop image and displayed on a monitor or the like. The screen information displayed by the application can be acquired by other applications (information recording application, etc.) as an image file or a video file consisting of a series of multiple images via an API of the OS (Operating System) or an API of the application to be displayed. The screen information of the desktop screen is information consisting of an image of the desktop screen generated by the OS, and can be acquired as an image file or a video file via the API of the OS in the same manner. The format of these image files may be bitmap, PNG, or other formats. The video file format may be MP4 or other formats.

[0099] The audio acquisition unit 15 acquires audio data received by the remote conference application 42 during the remote conference from the remote conference application 42. The audio acquisition unit 15 also passes the audio data acquired by the meeting device 60 to the remote conference application 42. Note that the audio acquired by the audio acquisition unit 15 does not include the audio collected by the terminal device 10, but only the audio data received during the remote conference. This is because the meeting device 60 separately collects the audio.

[0100] The device communication unit 16 communicates with the meeting device 60 using a USB cable or the like. The device communication unit 16 may communicate with the meeting device 60 via wireless LAN, Bluetooth (registered trademark), or the like. The device communication unit 16 receives a panoramic image and a speaker image from the meeting device 60, and transmits the audio data acquired by the audio acquisition unit 15 to the meeting device 60. The device communication unit 16 receives the audio data synthesized by the meeting device 60.

[0101] The recording control unit 17 combines the panoramic image and the speaker image received by the device communication unit 16 and the screen of the application acquired by the application screen acquisition unit 14 to create a combined image. The recording control unit 17 also creates a combined image video by connecting the repeatedly created combined images in chronological order, and creates a combined image video with audio by combining the synthesized audio data with the combined image video. The panoramic image and the speaker image may be combined by the meeting device 60. The recording control unit 17 may also store videos consisting of each image, such as the panoramic image, the speaker image, the application screen, and an image consisting of the panoramic image and the speaker image, in the storage service system 70 as separate video files. In this case, the panoramic video, the speaker video, the video of the application screen, and the combined video of the panoramic image and the speaker image may be called up at the time of viewing and displayed on one display screen.

[0102] The audio data processing unit 18 requests the information processing system 50 to convert the audio data combined with the combined image video extracted by the recording control unit 17, or the synthesized audio data received from the meeting device 60, into text data.

[0103] The combined image moving image is stored in the terminal device 10 during recording, and is then uploaded to the information processing system 50.

[0104] When the remote conference ends, the upload unit 20 transmits the combined image moving image to the information processing system 50.

[0105] The edit processing unit 21 executes editing of the combined image moving image (deleting a part, splicing, etc.) in response to a user operation.

[0106] The code analysis unit 22 detects a two-dimensional code or a barcode included in the panoramic image, and analyzes the two-dimensional code or the barcode to obtain conference participation information. The conference participation information includes information that the device is usable in the conference, device identification information of the electronic whiteboard 2 stored in the device information storage unit 3001, a conference ID (selected by the user), an IP address of the electronic whiteboard 2, and the like. The device identification information may be a serial number or a UUID (Universally Unique Identifier). The device identification information may be set by the user. The conference ID is assigned when the conference is booked, or when recording starts. The conference ID may be linked to a conference ID determined by the remote conference service system.

[0107] Fig. 10 shows video recording information stored in the information storage unit 1001. The video recording information has items such as a conference ID, a recording ID, an update date and time, a title, an upload, and a storage destination. When a user logs into the information processing system 50, the information recording application 41 downloads the conference information from the conference information storage unit 5001 of the information processing system 50. The conference ID and other information included in the conference information are reflected in the video recording information. The video recording information in Fig. 10 is held by a terminal device 10 operated by a certain user.

[0108] The conference ID is identification information for identifying a remote conference that has been held. The conference ID is assigned when a remote conference schedule is registered in the conference management system 9, or the information processing system 50 assigns a conference ID in response to a request from the information recording application 41. The conference management system 9 is a system for registering conference and remote conference schedules, a URL (conference link) for starting a remote conference, reservation information for devices to be used in the conference, and the like, and is a scheduler or the like connected from the terminal device 10 via a network. The conference management system 9 is also capable of transmitting the registered schedule, etc. to the information processing system 50.

[0109] The recording ID is identification information for identifying the combined image video recorded during a remote conference. The recording ID is assigned by the meeting device 60, but may be assigned by the information recording application 41 or the information processing system 50. A different recording ID is assigned to the same conference ID when recording was stopped in the middle of a remote conference but was resumed for some reason.

[0110] The update date and time is the date and time when the combined image video was updated (recording ended). If the combined image video was edited, it is the edit date and time.

[0111] The title is the name of the conference. It may be set when the conference is registered in the conference management system 9, or may be arbitrarily set by the user.

[0112] The "upload" indicates whether the combined image video has been uploaded to the information processing system 50 or not.

[0113] The destination indicates the location (URL or file path) where the combined image video and text data are saved in the storage service system 70. Therefore, the user can view the uploaded combined image video at will. The combined image video and text data are saved with separate file names following the URL, for example.

[0114] <<Meeting Devices>> Returning to Fig. 9, the meeting device 60 has a terminal communication unit 61, a panoramic image creation unit 62, a speaker image creation unit 63, a sound collection unit 64, and a voice synthesis unit 65. Each of these units of the meeting device 60 is a function or a means for performing a function that is realized when any of the components shown in Fig. 5 operates in response to an instruction from the CPU 611 in accordance with a program loaded from the ROM 612 to the DRAM 614.

[0115] The terminal communication unit 61 communicates with the terminal device 10 using a USB cable or the like. The terminal communication unit 61 may be connected not only by a wired cable but also by wireless LAN, Bluetooth (registered trademark), or the like as long as it can communicate with the terminal device 10.

[0116] The panoramic image creation unit 62 creates a panoramic image. The speaker image creation unit 63 creates a speaker image. The creation methods have been described with reference to Figures 6 and 7. The panoramic image creation unit 62 can obtain a two-dimensional code including conference participation information.

[0117] The sound collection unit 64 converts the sound picked up by the microphone 608 mounted on the meeting device 60 into audio data (digital). This allows the contents of speech made by users and participants at the base on the terminal device 10 side to be collected. The function of the sound collection unit 64 will be described in detail with reference to FIG. 11.

[0118] The voice synthesizer 65 synthesizes the voice transmitted from the terminal device 10 and the voice collected by the voice collector 64. As a result, the voice spoken at the other location 101 and the voice spoken at the own location 102 are combined.

[0119] 11 is a functional block diagram for explaining in detail the function of the sound collection unit 64. The sound collection unit 64 has an echo signal output unit 641, an echo signal acquisition unit 642, an echo component detection unit 644, an echo component amount calculation unit 643, a determination unit 645, and a control unit 646. The functions of the sound collection unit 64 may be the same on the other base side and on the own base side, but only the echo signal output unit 641 is used on the other base side.

[0120] The echo signal output unit 641 detects silent intervals from the magnitude of the output signals from the speaker 619 and microphone 608, and transmits a pseudo echo signal from the speaker 619 during the silent interval. The pseudo echo signal is, for example, a mosquito sound (ultrasonic waves in a frequency band of 16,000 Hz or more that are difficult for humans to hear). The output pseudo echo signal is output from the speaker 619 on the other base side and input to the microphone 608 on the other base side, and the echo signal acquisition unit 642 (on the other base side) receives this pseudo echo signal. 16,000 Hz is just an example, and any frequency that is difficult for humans to hear may be used. The echo signal output unit 641 is used on the other base side.

[0121] The pseudo echo signal acquired by the echo signal acquisition unit 642 is transmitted to the own base side via the remote conference service system. The pseudo echo signal passes through the speaker 619 on the own base side and is input to the microphone 608 on the own base side. The echo signal acquisition unit 642 (own base side) acquires the pseudo echo signal input to the microphone 608. The echo signal acquisition unit 642 performs echo cancellation by removing the audio data output by the speaker 619 from the audio data acquired by the microphone 608. Therefore, the pseudo echo signal acquired by the echo signal acquisition unit 642 has already been echo cancelled.

[0122] The echo component detector 644 detects the pseudo echo signal as an echo component. That is, since the pseudo echo signal has been echo-canceled, the pseudo echo signal is itself an echo component. The echo component detector 644 detects echo components of 16000 Hz or higher by subjecting the pseudo echo signal to spectrum analysis.

[0123] The echo component amount calculation unit 643 calculates the echo component amount E from the sound pressure of the echo component (power obtained by spectrum analysis) detected by the echo component detection unit 644. The echo component amount E may be, for example, a peak value or an effective value of the echo component obtained by spectrum analysis, or a normalized value of these.

[0124] The determination section 645 compares the amount of echo components E with a threshold value. The control section 646 automatically adjusts the level of noise reduction according to the result of the determination by the determination section 645. These will be described in detail later.

[0125] <<Information Processing System>> The information processing system 50 has a communication unit 51, an authentication unit 52, a screen generation unit 53, a communication management unit 54, a device management unit 55, and a text conversion unit 56. Each of these units in the information processing system 50 is a function or a means for performing a function that is realized when any of the components shown in Fig. 4 operates in response to an instruction from the CPU 501 in accordance with a program loaded from the HD 504 to the RAM 503. The information processing system 50 also has a storage unit 5000 constructed by the HD 504 etc. shown in Fig. 4. In the storage unit 5000, a conference information storage unit 5001, a recording information storage unit 5002, and an association information storage unit 5003 are constructed.

[0126] The communication unit 51 transmits and receives various information to and from the terminal device 10. For example, the communication unit 51 transmits a list of remote conferences to the terminal device 10 and receives a request for recognizing voice data from the terminal device 10.

[0127] The authentication unit 52 authenticates the user who operates the terminal device 10. For example, the authentication unit 52 authenticates the user based on whether or not authentication information (user ID and password) included in an authentication request received by the communication unit 51 matches authentication information stored in advance. Note that the authentication information may be an IC card number, biometric authentication information such as face or fingerprint, etc. Also, the authentication unit 52 may perform authentication using an external authentication system or an authentication method such as OAUTH.

[0128] The screen generation unit 53 generates screen information to be displayed by the terminal device 10. When the terminal device 10 executes a native application, the screen information is held by the terminal device 10, and the information to be displayed is transmitted in XML or the like. When the terminal device 10 executes a Web application, the screen information is generated by HTML, XML, CSS (Cascade Style Sheet), JavaScript (registered trademark), or the like.

[0129] The communication management unit 54 acquires information related to the remote conference from the conference management system 9 using each user's account or a system account assigned to the information processing system 50. The communication management unit 54 stores the conference information of the reserved conference in the conference information storage unit in association with the conference ID. The communication management unit 54 also acquires conference information that users belonging to the tenant have permission to view. Since a conference ID is set for the conference, the remote conference and recorded information are associated with each other by the conference ID.

[0130] When the equipment management unit 55 receives the equipment identification information of the electronic whiteboard 2 or the meeting device 60, the equipment management unit 55 associates the electronic whiteboard 2 and the meeting device 60 used in the conference with each other for the conference, and stores the association information in the association information storage unit 5003. Therefore, the conference ID, the equipment identification information of the electronic whiteboard 2, and the equipment identification information of the meeting device 60 are associated with each other. Since the combined image video is also associated with the conference ID, the object data and the combined image video are also associated with each other. Furthermore, when the recording ends (when the conference ends), the equipment management unit 55 deletes the association from the association information storage unit 5003.

[0131] The text conversion unit 56 converts the voice data requested by the terminal device 10 to be converted into text data into text data using an external voice recognition service system 80. The text conversion unit 56 may also perform the conversion itself.

[0132] FIG. 12 is an example of conference information stored in the conference information storage unit 5001 managed by the communication management unit 54. The communication management unit 54 can use the above account to obtain a list of remote conferences for which the user belonging to the tenant has viewing authority. In this embodiment, a remote conference is taken as an example, but the list of remote conferences also includes conferences held in only one conference room. Viewing authority may be granted to the conference information managed by the communication management unit 54 directly from the information recording application of the terminal device 10. Furthermore, the information of remote conferences for which the user belonging to the tenant has viewing authority includes information of conferences created by the user and information of conferences for which the user has been given viewing authority by other users.

[0133] The conference information is managed by a conference ID and is associated with participants, a title (conference name), a start date and time, an end date and time, a location, etc. These are examples of conference information, and the conference information may include other information.

[0134] Participant items are the participants in a meeting.

[0135] The title item indicates the content of the meeting, such as the meeting name or the agenda.

[0136] The Start Date and Time field is the date and time when the meeting is scheduled to start.

[0137] The End Date and Time item is the date and time when the conference is scheduled to end.

[0138] The location item is the location where the meeting is held, such as a conference room, a branch office name, or a building.

[0139] The item "electronic whiteboard" is the device identification information of the electronic whiteboard 2 used in the meeting.

[0140] The meeting device item is the device identification information of the meeting device 60 used in the conference.

[0141] As shown in FIGS. 10 and 12, a combined image video recorded at a conference is identified by a conference ID.

[0142] The recording information stored in the recording information storage unit 5002 may be the same as that in Fig. 10. However, the information processing system 50 has a list of combined image videos recorded by all users belonging to the tenant. The user may input desired storage destination information (path information such as a URL of a cloud storage system) on a user setting screen of the information recording application 41 of the terminal device 10, and the desired storage destination information may be stored in the recording information storage unit 5002.

[0143] 13 shows association information in which the device identification information of the electronic whiteboard 2 and the device identification information of the meeting device 60 are associated with the conference ID, which is stored in the association information storage unit 5003. This association information is held from the time of joining the conference until the end of the conference (leaving the room).

[0144] <<Electronic whiteboard>> 14 is a functional block diagram explaining the functions of the electronic whiteboard 2 by dividing them into blocks. The electronic whiteboard 2 has a contact position detection unit 31, a drawing data generation unit 32, a data recording unit 33, a display control unit 34, and a communication unit 35. Each function of the electronic whiteboard 2 is a function or means realized by operating any of the components shown in FIG. 8 by an instruction from the CPU 401 according to a program loaded from the SSD 404 onto the RAM 403.

[0145] The contact position detection unit 31 detects the coordinates of the position where the electronic pen 490 contacts the contact sensor 414. The drawing data generation unit 32 acquires the coordinates where the tip of the electronic pen 490 contacts from the contact position detection unit 31. The drawing data generation unit 32 generates stroke data by connecting the sequence of coordinate points by interpolating them.

[0146] The display control unit 34 displays the handwritten data, character strings converted from the handwritten data, and menus for user operations on the display.

[0147] The data recording unit 33 stores handwritten data written on the electronic whiteboard 2, shapes obtained by converting handwritten data into circles, triangles, and the like, stamps such as "Done," PC screens, files, and the like in the object information storage unit 3002. Handwritten data, character strings (including shapes), images such as PC screens, files, and the like are handled as objects. For handwritten data, a group of stroke data becomes one object according to a time division due to an interruption in handwriting or a distance division due to a difference in handwriting location.

[0148] The communication unit 35 is connected to WiFi or a LAN and communicates with the information processing system 50. The communication unit 35 transmits object information to the information processing system 50, receives object information stored in the information processing system 50 from the information processing system 50, and displays the object information on the display 480.

[0149] The electronic whiteboard 2 also has a storage unit 3000 constructed by the SSD 404 etc. shown in Fig. 8. In the storage unit 3000, a device information storage unit 3001 and an object information storage unit 3002 are constructed.

[0150] FIG. 15 shows the device identification information and the like stored in the device information storage unit 3001.

[0151] The device identification information is the device identification information of the electronic whiteboard 2.

[0152] The IP address is an IP address for other devices to connect to the electronic whiteboard 2 via a network.

[0153] The passcode is used to authenticate other devices when they connect to the electronic whiteboard 2.

[0154] 16 is a diagram explaining the object information stored in the object information storage unit 3002. The object information is information for managing objects displayed by the electronic whiteboard 2. The object information is transmitted to the information processing system 50 and used as minutes of a meeting. If an electronic whiteboard 2 is located at another location when a remote conference is held, the object information is shared.

[0155] The conference ID is identification information of the conference notified by the information processing system 50.

[0156] · Object ID is identification information that identifies an object.

[0157] Type is the type of object, and can be handwritten, character, figure, image, etc. Handwritten is stroke data (a sequence of coordinate points). Character is a character string (character code) converted from handwritten data. Character strings are sometimes called text data. Figures are geometric shapes such as triangles and squares converted from handwritten data. Images are image data such as Jpeg, PNG, TIFF, etc. imported from a PC or the Internet.

[0158] Each screen of the electronic whiteboard 2 is called a page. The item on a page is its page number.

[0159] The coordinates indicate the position of an object relative to a predetermined origin of the electronic whiteboard 2. The position of an object is, for example, the upper left vertex of the bounding rectangle of the object. The coordinates are expressed, for example, in pixel units of the display.

[0160] Size is the width and height of the object's bounding rectangle.

[0161] <Example of microphone and speaker installation in a meeting device> 17 shows a conceptual diagram of a microphone 608 and a speaker 619 mounted on the meeting device 60. The meeting device 60 connected to the terminal device 10 has an imaging unit 601 capable of capturing panoramic images as described above, a microphone 608, and a speaker 619. The meeting device 60 transmits image data captured by the imaging unit 601 and audio data acquired by the microphone 608 to the terminal device 10. The meeting device 60 also receives audio data (acquired from the remote conference application 42) transmitted by the terminal device 10, and outputs audio from the speaker 619.

[0162] The meeting device 60 has an external device connection I / F that can be connected to the terminal device 10 via a wired cable in order to transmit and receive audio data and image data. Alternatively, the meeting device 60 has a wireless communication I / F that can wirelessly communicate with the terminal device 10. The meeting device 60 may be directly connected to an in-house server, a cloud system, a Web conference system, or the like (not limited to these) instead of the terminal device 10. Possible examples of the external device connection I / F include USB, HDMI (registered trademark), DVI, DISPLAYPORT, and Ethernet (registered trademark), and possible examples of the wireless communication I / F include Wi-Fi (registered trademark), Bluetooth (registered trademark), but are not limited to these.

[0163] <Principles of echo generation and echo cancellers> The principle of echo generation in a remote conference will be described with reference to Fig. 18. Fig. 18 is a diagram for explaining the principle of echo generation. In a remote conference, a microphone 608 and a speaker 619 are used on the remote base side and the own base side, respectively.

[0164] (1) The voice of the user at the other base is picked up by the microphone 608, and sound based on the voice data transmitted to the user's base is output from the speaker 619 at the user's base.

[0165] (2) This sound is input to the microphone 608 at the own location, and the voice data is transmitted to the other location. The speaker 619 at the other location outputs sound based on the voice data.

[0166] Therefore, in stages (1) and (2), the voice of the user at the other base returns to the other base, and the original voice is output from the other base's speaker 619. The voice data of the user at the other base that returns to the other base is an echo component, and this phenomenon is called an echo.

[0167] (3) The echo canceller removes the sound data output by the speaker 619 at the own base from the sound data input to the microphone 608 at the own base. This allows the meeting device 60 to reduce the volume of the user's voice returning to the other base.

[0168] However, in the voice data sent from the other base station, the amount of echo components is reduced by the echo canceller, but it may not be possible to reduce it to zero, which may result in an echo (the speech of the user at the other base station) being output from the speaker 619 at the other base station.

[0169] 19 is a diagram explaining two echo cancellation methods (echo suppressor method and adaptive echo canceller method). The echo suppressor method is a method that forcibly prevents echo from occurring by turning off the microphone 608 on the home base when the voice from the other base is being output from the speaker 619 on the home base. Since the route by which the sound returns is blocked, echo can be completely suppressed, but there is a problem that the home base and the other base cannot speak at the same time because no sound can be picked up because the microphone 608 on the home base is turned off.

[0170] On the other hand, the adaptive echo canceller method estimates the echo picked up by the microphone 608 on the local base side and removes only the echo component, and while it has the advantage of allowing simultaneous speech, it cannot completely suppress the echo. This embodiment is a technology that has great advantages when an adaptive echo canceller is used.

[0171] <Example of sound processing unit configuration> Fig. 20 shows the audio processing and functional configuration performed by the sound processing unit 609. Each function of the sound processing unit 609 may be realized by hardware such as an IC, by software, or by a mixture of both. The upper part of Fig. 20 shows the processing from microphone input to microphone output, and the lower part shows the processing from speaker input to speaker output.

[0172] The microphone input signal (audio data) input to the microphone 608 has its tone adjusted by the equalizer 301. The equalizer 301 can adjust the gain for each frequency, and adjusts the waveform to achieve the desired frequency characteristics. Next, the sine gain unit 302 adjusts (e.g. amplifies) the magnitude of the signal. After that, the adaptive echo canceller 303 suppresses echoes by removing the speaker output signal from the microphone input signal. In other words, the adaptive echo canceller 303 creates a signal that is the same as the speaker output signal and removes it from the microphone input signal.

[0173] Furthermore, the beam forming unit 304 identifies the direction of the signal inside the microphone and reduces the sound input from other directions. The noise reduction unit 305 (an example of a noise reduction function) sets the amount of reduction of external noise and adjusts the level according to the usage environment. The noise reduction unit 305 reduces and removes only the noise components at the set level using an existing method. The existing method is, for example, a low-pass filter or BSS (blind source separator, a process that separates human voices from other sounds and reduces only the other sounds) and performs noise reduction using one or more functions. The auto gain control (AGC) unit 306 adjusts the input and output ratio for each signal level to reduce loud sounds while increasing quiet sounds. Finally, the Sout gain unit 307 adjusts the microphone input signal to the desired signal level.

[0174] Similarly, for the speaker input signal, an equalizer 308 adjusts the tone, an auto gain control unit 309 adjusts the input / output ratio for each signal level, and an Rout gain unit 310 adjusts the signal level to the target level. The speaker input signal (hence the speaker output signal) after adjustment by the Rout gain unit 310 is input to the microphone-side adaptive echo canceller 303 described above, where adjustments are made to suppress echoes from two pieces of information: the microphone input signal and the speaker output signal.

[0175] Generally, the longer the echo is perceived by the human ear and the higher the signal level (sound pressure), the worse the echo is heard, so countermeasures are often taken by shortening the time, lowering the signal level, or both. Conventional echo suppression techniques can be broadly divided into two types: changing the parameters of an adaptive echo canceller and changing the gain. Changing the parameters of an adaptive echo canceller is often mainly a change in the time constant, and suppression is performed in the direction of shortening the echo time. There is no need to adjust the gain (speaker or microphone volume), and the sound heard by the user does not change, making it easy to hear, but the suppression effect is small and may not be sufficient. On the other hand, changing the gain changes the sound heard by the user, making it difficult to hear, but reduces the amount of echo components, so the echo suppression effect is relatively large.

[0176] 20, the processes capable of adjusting gain are the microphone-side equalizer 301, Sin gain section 302, auto gain control section 306, Sout gain section 307, speaker-side equalizer 308, auto gain control section 309, and Rout gain section 310. However, since fluctuations occur in the gain (volume) and frequency characteristics (tone) of the sound heard by the user, there is a risk that a sound that is difficult to hear will be output. This embodiment is configured to suppress echoes by controlling the level of noise reduction rather than gain.

[0177] <Echo Suppression in the Present Embodiment> In this embodiment, echo is reduced as follows: The meeting device 60 at the other site detects a silent section from the magnitude of the output signals from the speaker 619 and the microphone 608, and outputs a pseudo echo signal from the speaker 619 of the meeting device 60 at the other site during the silent section.

[0178] The output pseudo echo signal is picked up by the microphone 608 of the meeting device 60 at the other base and transmitted to the terminal device 10 at the own base via the terminal device 10 at the other base. The terminal device 10 at the own base transmits the pseudo echo signal to the meeting device 60 at the own base. The speaker 619 of the meeting device 60 at the own base outputs the pseudo echo signal, which is picked up by the microphone 608 of the meeting device 60 at the own base.

[0179] The meeting device 60 at the own base calculates the echo component amount E from the sound pressure of the pseudo echo signal. The meeting device 60 automatically adjusts the noise reduction level according to the echo component amount E, so that the echo component amount can be reduced.

[0180] The reason why the noise reduction level is effective in reducing the amount of echo components is as follows: 1. Improved accuracy of adaptive filters (low sound pressure during speech is less likely to be mistaken for noise, reducing choppy sound) 2. Echo components are removed as part of the noise (if the amount of echo components is small, they are easily removed as noise) There are two types:

[0181] <Regarding processing or operation> First, the flow of voice data in a remote conference will be described with reference to Fig. 21. Fig. 21 is a sequence diagram explaining the flow of voice data transmission from another location to the own location in a remote conference. Image data is also transmitted together with the voice data, but the transmission of image data is omitted in the figure.

[0182] S11: The microphone 608 of the meeting device 60 picks up remarks from participants at other locations.

[0183] S12: The terminal communication unit 61 of the meeting device 60 transmits the audio data to the terminal apparatus 10.

[0184] S13: The voice acquisition unit 15 of the terminal device 10 at the other site passes the voice data to the remote conference application 42. The remote conference application 42 transmits the voice data to the remote conference service system 90.

[0185] S14: The remote conference service system 90 transmits the received voice data to all the terminal devices 10 participating in the same conference. In FIG. 21, it is assumed that the voice data is transmitted to the terminal device 10 at the own base.

[0186] S15: The remote conference application 42 of the terminal device 10 at the own base receives the voice data. The voice acquisition unit 15 acquires the voice data and passes it to the device communication unit 16. The device communication unit 16 transmits the voice data to the meeting device 60 at the own base.

[0187] S16: The terminal communication unit 61 of the meeting device 60 at the own base receives the audio data and outputs it from the speaker 619.

[0188] In such a flow of audio data, the meeting device 60 at the other site and the meeting device 60 at the own site reduce echoes in the following manner.

[0189] First, the echo signal output unit 641 of the meeting device 60 at the other site detects a silent period from the magnitude of the output signals of the speaker 619 and microphone 608 of the meeting device 60 at the other site.

[0190] Fig. 22 is a diagram for explaining the magnitude of the output signal of the speaker 619 and the input signal of the microphone 608. Fig. 22(a) and (b) respectively show examples of sound pressure information of sounds A and B output by the speaker 619, or sound pressure information of sounds A and B acquired by the microphone 608. The sound pressure is the amplitude of the waveforms in Fig. 22(a) and (b), and is acquired by converting the input voltage VIN into decibel full scale (dBFS) using equation (1). In other words, the decibel full scale is the sound pressure. The sound pressures in Fig. 22(a) and (b) are calculated in this manner.

number

[0191] The peak value is a method in which the peak value (maximum value) of an audio waveform is used as a representative value of the sound pressure information of a certain audio data. The effective value (RMS value) is a method in which the effective value VIN-rms of the waveform of the input voltage VIN of a certain audio is converted to sound pressure (dBFS) using the above formula (1) and used as a representative value of the sound pressure information. The effective value can be calculated using formula (2). Figure 22(c) shows the peak values ​​and effective values ​​of audio A and B.

number

[0192] When a silent period continues for a certain period or more, the echo signal output unit 641 on the other base side outputs a pseudo echo signal from the speaker 619 (S101). The pseudo echo signal needs to be a signal that is as difficult to hear as possible for the human ear so as not to hinder conversation, so a sound source with varying sound pressure is unsuitable. For this reason, the pseudo echo signal is a sine wave of 16000 Hz, and the transmission time is about 0.1 seconds. When the echo signal output unit 641 on the other base side detects a silent period, it is preferable to transmit a notice to the meeting device 60 on the own base side that a pseudo echo signal will be transmitted, together with or before the pseudo echo signal.

[0193] It is preferable that the meeting device 60 at the own base also detects silent intervals. This allows the meeting device 60 at the own base to predict when it is time to adjust the noise reduction level (when a pseudo echo signal is transmitted from the meeting device 60 at the other base). The meeting device 60 at the own base only needs to detect the echo component when it detects a silent interval and control the noise reduction level according to the amount of the echo component, thereby reducing the processing load.

[0194] The output pseudo echo signal is input to the microphone 608 of the meeting device 60 at the other site, and is acquired by the echo signal acquisition unit 642 (an example of a first echo signal acquisition unit). The meeting device 60 at the other site outputs the pseudo echo signal from the speaker 619 and the microphone 608 acquires the pseudo echo signal in the state where the echo is canceled at the other site, so that the meeting device 60 at the own site can analyze the pseudo echo signal. In other words, the meeting device 60 at the other site can transmit the pseudo echo signal under conditions similar to those when speaking. Note that the conditions for echo generation during a conference require a speaker and microphone at the own site and a speaker and microphone at the other site. If any one of them is missing or muted, no echo is generated, and in that case the component of the pseudo echo signal also becomes 0. In order to correctly extract the pseudo echo component, it is necessary to pass through the speaker and microphone at the other site. Note that, for example, when the meeting device 60 at the other site is muted, the echo signal output unit 641 at the other site may directly transmit the pseudo echo signal to the own site.

[0195] The terminal communication unit 61 of the meeting device 60 at the other base transmits the pseudo echo signal to the terminal device 10 at the other base. The voice acquisition unit 15 of the terminal device 10 at the other base passes the pseudo echo signal to the remote conference application 42. The remote conference application 42 transmits the pseudo echo signal to the remote conference application 42 running on the terminal device 10 at the own base via the remote conference service system 90. With the above, the terminal device 10 at the own base receives the pseudo echo signal (S102).

[0196] The remote conference application 42 of the terminal device 10 at the own base receives the pseudo echo signal and passes it to the voice acquisition unit 15. The device communication unit 16 that receives the pseudo echo signal from the voice acquisition unit 15 transmits the pseudo echo signal to the meeting device 60 at the own base. This meeting device 60 outputs the pseudo echo signal from the speaker 619, so that the pseudo echo signal is input to the microphone 608. The echo signal acquisition unit 642 (an example of a second echo signal acquisition unit) of the meeting device 60 at the own base acquires the pseudo echo signal input to the microphone 608.

[0197] The meeting device 60 at the own base outputs the signal to the speaker 619 and the microphone 608 picks up the signal, so that the own base also performs echo cancellation on the pseudo echo signal, allowing the echo component detection unit 644 to detect the echo component of the echo-canceled pseudo echo signal.

[0198] The meeting device 60 at the own site may generate a pseudo echo signal and output it from the microphone 608 instead of receiving the pseudo echo signal from the meeting device 60 at the other site.

[0199] The echo component detection unit 644 detects the echo component by performing a spectrum analysis on the pseudo echo signal and extracting a pseudo echo signal of 16000 Hz or more. Since it is a silent section, the sound data of 16000 Hz or more is the echo component as it is. The echo component amount calculation unit 643 calculates the echo component amount E from the sound pressure of the pseudo echo signal detected by the echo component detection unit 644 (S103). The echo component amount calculation unit 643 sets the effective value (RMS value) or peak value of the spectrum-analyzed power as the echo component amount E. The echo component amount E may be normalized to a range of 0 to 100, for example.

[0200] The determination unit 645 determines whether the echo component amount E is zero (S104). Note that zero may include a value that is so small that it can be regarded as zero.

[0201] If the determination in step S104 is Yes, since there is almost no echo component amount, the control unit 646 reduces the noise reduction level (S105). In other words, it becomes difficult to remove noise. As described in FIG. 24, by reducing the noise reduction level, the echo component amount increases, but it becomes difficult to remove small sound pressure during speech as noise. The amount of reduction may be a fixed value, or a fixed ratio to the current noise reduction level. Alternatively, the control unit 646 may determine the amount of reduction based on a previously prepared table that associates the noise reduction level with the amount of reduction.

[0202] If the determination in step S104 is No, the determination unit 645 determines whether the echo component amount E is greater than a threshold value E1 (S106).

[0203] If the determination in step S106 is Yes, the amount of echo components is large, so the control unit 646 increases the noise reduction level (S107). That is, it becomes easier to remove noise. As described in FIG. 24, by increasing the noise reduction level, the amount of echo components can be reduced, making it easier to suppress echoes. The amount of increase may be a fixed value, or a fixed ratio to the current noise reduction level. Alternatively, the control unit 646 may determine the amount of increase based on a previously prepared table that associates the noise reduction level with the amount of increase.

[0204] If the determination in step S106 is No, the amount of echo components is not zero but is not so large, so that the control unit 646 maintains (does not change) the noise reduction level.

[0205] In this embodiment, since there are at least two levels of noise reduction, an appropriate noise reduction level can be maintained according to the amount of echo components E. If the noise reduction level is too high, the effect of noise reduction is too large, and the area with low sound pressure during speech (for example, the end of a word) is mistakenly recognized as noise and the noise is removed together with the noise, which has the disadvantage that the sound tends to sound choppy. On the other hand, if the noise reduction level is too low, the effect of noise reduction is small and echo suppression cannot be performed. Since the meeting device 60 of this embodiment is configured to control the noise reduction level in real time, it can always be controlled to the optimal noise reduction level at the present time. In addition, the optimal noise reduction level does not remove speech with low sound pressure and can also reduce noise, so echo can be appropriately reduced.

[0206] In addition, during silent sections, echo signals that are difficult for humans to hear and are above 16,000 Hz are emitted, making it possible to calculate the amount of echo components without disrupting conversation.

[0207] In this way, this embodiment is configured to suppress echoes by controlling the noise reduction level rather than the gain, so there is no fluctuation in the gain (volume) and frequency characteristics (tone) of the sound heard by the user, and it is possible to provide sound that is easy to hear.

[0208] In the process of FIG. 23, the meeting device 60 at the own base controls the level of noise reduction, but it is preferable that the meeting device 60 at the other base performs the same process and controls the level of noise reduction.

[0209] 23, the meeting device 60 controls the noise reduction level in two stages using one threshold E1, but the noise reduction level may be controlled in three or more stages using two or more thresholds. Since the noise reduction level can be controlled more finely, the risk of raising or lowering the noise reduction level too much can be reduced.

[0210] <Relationship between the amount of echo components during speech and the noise reduction level> Fig. 24 is a diagram showing the relationship between the noise reduction level and the magnitude of the echo component amount in several speech intervals. A speech interval is a time period (section) during which a participant speaks. Fig. 24 shows a case where the noise reduction level is set to two levels. In each speech interval, a bar graph shows the magnitude of the echo component amount when the noise reduction level is controlled to -5 dB and -20 dB.

[0211] As shown in Fig. 24, it can be seen that the higher the noise reduction level (the smaller the number with a minus sign = the larger the absolute value), the greater the effect of suppressing the amount of echo components. In other words, the magnitude of the amount of echo components can be reduced by controlling the noise reduction level.

[0212] On the other hand, as is clear from a comparison of speech sections A and D, the difference in noise reduction level is not proportional to the reduction in the amount of echo components. In speech section A, even if the noise reduction level differs from -5 dB to -20 dB, there is not much change in the amount of echo components. In speech section D, the amount of echo components changes significantly when the noise reduction level differs from -5 dB to -20 dB. In other words, controlling the noise reduction level does not simply remove the echo components as noise, but can be expected to have the effect of appropriately reducing the echo.

[0213] <Recording during the meeting> Next, several screens displayed by the terminal device 10 during a remote conference will be described with reference to Fig. 25 to Fig. 28. Fig. 25 shows an initial screen 200 after login displayed by the information recording application 41 running on the terminal device 10. The user of the terminal device 10 connects the information recording application 41 to the information processing system 50. When the user inputs authentication information and succeeds in logging in, the initial screen 200 of Fig. 25 is displayed.

[0214] The initial screen 200 has a fixed display button 201, a change to front button 202, a panoramic image 203, one or more speaker images 204a to 204c (hereinafter, referred to as speaker images 204 when no distinction is made), and a start recording button 205. If the meeting device 60 has already started up and is capturing images of the surroundings at the time of login, the panoramic image 203 and speaker image 204 created by the meeting device 60 are displayed on the initial screen 200. Therefore, the user can decide whether or not to start recording while viewing these. If the meeting device 60 is not started up (not capturing images), the panoramic image 203 and speaker image 204 are not displayed.

[0215] The information recording application 41 may display speaker images 204 of all participants based on all faces detected from the panoramic image 203, or may display only the speaker images 204 of the N most recent speakers. Fig. 25 shows an example in which speaker images 204 of up to three people are displayed. Until a participant speaks, there may be no speaker image 204 (the speaker images are increased one by one according to the speech), or three speaker images 204 of participants in a predetermined direction may be displayed (the speaker images are replaced according to the speech).

[0216] When no one is speaking, such as immediately after the meeting device 60 is started, a predetermined direction (0°, 120°, 240°, etc.) within the horizontal 360° is created as the speaker image 204. If a fixed display, which will be described later, is set, the setting of the fixed display takes priority.

[0217] The fixed display button 201 is a button for the user to perform an operation of fixing a certain area of ​​the panoramic image 203 in close-up as a speaker image 204 .

[0218] The front change button 202 is a button that allows the user to perform an operation to change the front of the panoramic image 203 (as a panoramic image covers 360° horizontally, the right and left edges face the same direction). The user can slide the panoramic image 203 left and right with a pointing device to determine which participant will be shown in front. The user's operation is transmitted to the meeting device 60, which then changes the angle of the front out of the 360° horizontal direction, creates a panoramic image, and transmits it to the terminal device 10.

[0219] When the user presses the record start button 205, the information recording application 41 displays the recording setting screen 210 shown in FIG.

[0220] 26 is an example of a recording setting screen 210 displayed by the information recording application 41. In the recording setting screen 210, the user can set whether to record (whether to include in recording) the panoramic image and speaker image created by the meeting device 60, and the desktop screen or the screen of a running application of the terminal device 10. If the information recording application 41 does not record either the panoramic image and speaker image, or the desktop screen or the screen of a running application, only the audio (audio output by the terminal device 10+audio collected by the meeting device 60) is recorded.

[0221] The camera toggle button 211 is a button for switching on and off the recording of the panoramic image and the speaker image created by the meeting device 60. The camera toggle button 211 may be set to record the panoramic image and the speaker image separately.

[0222] The PC screen toggle button 212 is a button for switching on and off recording of the desktop screen of the terminal device 10 and the screen of an application running on the terminal device 10. When the PC screen toggle button 212 is in the on state, the desktop screen is recorded.

[0223] If the user wants to record the screen of an application, the user further selects the application in the application selection field 213. The application selection field 213 displays the names of applications currently being executed by the terminal device 10 in a pull-down format. The user can select the application to be recorded. The information recording application 41 acquires the application name from the OS. The information recording application 41 can display only those applications having a UI (screen) among the currently executing applications. The remote conference application 42 may be included in the selected applications. Therefore, the information recording application 41 can record materials displayed in the remote conference application 42 and participants at each location as videos. In addition, the applications displayed in the pull-down menu are various applications currently being executed on the terminal device, such as a presentation application, a word processing application, a spreadsheet application, an application for creating and editing materials such as documents, a cloud electronic whiteboard application, and a web browser application. Therefore, the user can flexibly select the screen of an application to be included in the combined image video.

[0224] Furthermore, when recording on an app basis, the user can select multiple apps, and the information recording app 41 can record the screens of all the selected apps.

[0225] When both the camera toggle button 211 and the PC screen toggle button 212 are off, the video recording confirmation window 214 displays "Only audio will be recorded." This audio is the audio output by the terminal device 10 (audio received by the remote conference application 42 from the other location 101) and the audio collected by the meeting device 60. In other words, as long as a remote conference is being held, the audio of the remote conference application 42 and the audio of the meeting device 60 are saved regardless of the recording of images. However, the user may be able to selectively stop saving the audio of the remote conference application 42 and the audio of the meeting device 60 by user settings.

[0226] The combined image moving image is recorded as follows according to the on / off combination of the camera toggle button 211 and the PC screen toggle button 212. The combined image moving image is also displayed in real time in the recorded content confirmation window 214.

[0227] When the camera toggle button 211 is on and the PC screen toggle button 212 is off, the panoramic image captured by the meeting device 60 and the speaker image are displayed in the recorded content confirmation window 214 .

[0228] When the camera toggle button 211 is off and the PC screen toggle button 212 is on (the screen is also selected), the recording content confirmation window 214 displays the desktop screen or the screen of the selected application.

[0229] When the camera toggle button 211 is on and the PC screen toggle button 212 is on, the panoramic image and speaker image captured by the meeting device 60, as well as the desktop screen and the screen of the selected application are displayed side by side in the recording content confirmation window 214.

[0230] Therefore, there are cases where the panoramic image, the speaker image, and the app screen are not recorded, or cases where the panoramic image, the speaker image, and the app screen are not recorded at all, but in this embodiment, for convenience, the image created by the information recording application 41 is referred to as a combined image video.

[0231] Furthermore, the recording setting screen 210 has a check box 215 together with a message saying "Automatically transcribe after uploading recording." The recording setting screen 210 also has a start recording now button 216. When the user checks the check box 215, text data obtained by converting comments made during the remote conference is attached to the recorded video. In this case, after recording ends, the information recording application 41 uploads the audio to the information processing system 50 together with a request to convert it into text data. Furthermore, when the user presses the start recording now button 216, the recording in progress screen 220 in FIG. 27 is displayed.

[0232] FIG. 27 is an example of a recording screen 220 displayed by the information recording application 41 during recording. Note that the explanation of FIG. 27 will mainly focus on the differences from FIG. 25. The recording screen 220 displays a combined image video being recorded in real time under the conditions set by the user on the recording setting screen 210. The recording screen 220 in FIG. 27 is when the camera toggle button 211 is on and the PC screen toggle button 212 is off, and displays a panoramic image 203 and a speaker image 204 (both of which are videos) created by the meeting device 60. The recording screen 220 displays a recording icon 225, a pause button 226, and an end recording button 227.

[0233] The pause button 226 is a button for stopping recording, and after stopping, restarting of recording is also accepted. The end recording button 227 is a button for ending recording. The pause button 226 does not switch the recording ID, but the end recording button 227 switches the recording ID. After pausing, when restarting recording, the user can also set the recording conditions set on the recording setting screen 210 again. In this case, the information recording application 41 may create multiple recording files each time recording is stopped (for example, by pressing the end recording button 227), or may combine multiple files so that they are continuous as one video (for example, by pressing the pause button 226). Furthermore, when the information recording application 41 plays back a combined image video, the multiple recording files may be played back continuously as one video.

[0234] Furthermore, the recording in progress screen 220 has an Get Information From Calendar button 221, a conference name column 222, a time column 223, and a location column 224. The Get Information From Calendar button 221 is a button for the user to obtain conference information from the conference management system 9. When the Get Information From Calendar button 221 is pressed, the information recording application 41 obtains a list of conferences that the user has viewing authority from the information processing system 50, and displays it. The user selects a remote conference to be held from the conference list. As a result, the conference information is reflected in the conference name column 222, the time column 223, and the location column 224. The title of the conference information is reflected in the conference name column 222, the start time and end time are reflected in the time column 223, and the location is reflected in the location column 224. Furthermore, the conference information and the recording information in the conference management system are associated with each other by a conference ID.

[0235] Once the remote conference is over and the user stops recording, a combined image video with audio is created.

[0236] 28 is an example of the conference list screen 230 displayed by the information recording application 41. The conference list screen 230 is a list of conferences, and can display a list of recorded information recorded in remote conferences. It also includes conferences that were held only in a certain conference room, regardless of whether they were remote conferences. The conference list screen 230 displays conference information that the logged-in user has viewing authority for in the conference information storage unit 5001. Information on the video recording information stored in the information storage unit 1001 may be further integrated.

[0237] The conference list screen 230 is displayed when the user selects the conference list tab 231 on the initial screen 200 in Fig. 25. The conference list screen 230 displays a list 236 of recorded information that the user has viewing authority for. The conference creator (person who creates the minutes) can set viewing authority for participants. The conference list may be a list of stored recorded information or a list of conference schedules and conference data.

[0238] The conference list screen 230 has the following items: check box 232 , update date and time 233 , title 234 , and status 235 .

[0239] The checkboxes 232 are used to accept the selection of recorded files. The checkboxes 232 are used when the user wishes to delete recorded files collectively.

[0240] The update date and time 233 indicates the start and end times of recording the combined image video. If the video has been edited, the edit date and time may be used.

[0241] The title 234 is the title of the meeting (agenda, etc.). This may be transcribed from the meeting information or may be set by the user.

[0242] The status 235 indicates whether or not the combined image video has been uploaded to the information processing system 50. If it has not been uploaded, "Local PC" is displayed, and if it has been uploaded, "Uploaded" is displayed. If it has not been uploaded, an upload button is displayed. If there is a combined image video that has not been uploaded, it is preferable that the information recording application 41 automatically uploads it when the user logs in to the information processing system 50.

[0243] When the user selects an arbitrary title or the like from the list 236 of combined image videos with a pointing device, the information recording application 41 displays a recording playback screen, but this is omitted in this embodiment. On the recording playback screen, it is possible to play back the combined image video.

[0244] It is desirable for the user to be able to narrow down the conferences based on the update date and time, title, keywords, etc. Furthermore, when there are a large number of conferences displayed and it is difficult to find the relevant conference, it is desirable for the user to be able to input a search term to narrow down the recorded information based on the words contained in the remarks made at the conference or the title, etc. The search function allows the user to find the desired recorded information in a short time even when there is a large amount of recorded information. Furthermore, on the conference list screen, the user may be able to sort the conferences by update date and time or title.

[0245] <<Recording operation or processing>> 29 is an example of a sequence diagram showing a procedure for recording a panoramic image, a speaker image, and an application screen by the information recording application 41. Conference participation and mute control have already been completed.

[0246] S201: The user operates the remote conference application 42 to start a remote conference. Here, it is assumed that the remote conference applications 42 of the own base 102 and the other base 101 start a remote conference. The remote conference application 42 of the own base 102 transmits an image captured by the camera of the meeting device 60 and an audio collected by the microphone 608 to the remote conference application 42 of the other base 101. The remote conference application 42 of the other base 101 displays the received image on a display and outputs the received audio from a speaker 619. Similarly, the remote conference application 42 of the other base 101 transmits an image captured by the camera of the meeting device 60 and an audio collected by the microphone 608 to the remote conference application 42 of the own base 102. The remote conference application 42 of the own base 102 displays the received image on a display and outputs the received audio from a speaker 619. Each remote conference application 42 repeats this to realize the remote conference.

[0247] S202: The user performs settings related to recording on the recording setting screen 210 of the information recording application 41 shown in Fig. 26. The operation acceptance unit 12 of the information recording application 41 accepts the settings. Here, it is assumed that the camera toggle button 211 and the PC screen toggle button 212 are both on.

[0248] S203: When the user operates to start recording, the recording control unit 17 of the information recording application 41 starts recording.

[0249] S204: The application screen acquisition unit 14 of the information recording application 41 requests the application screen selected by the user (more specifically, the application screen acquisition unit 14 acquires the application screen via the OS). In FIG. 29, the application selected by the user is the remote conference application 42.

[0250] S205: The recording control unit 17 of the information recording application 41 notifies the meeting device 60 of the start of recording via the device communication unit 16. When notifying, the recording control unit 17 may also notify that the camera toggle button 211 is on (request for a panoramic image and a speaker image). Regardless of the presence or absence of a request, the meeting device 60 transmits the panoramic image and the speaker image to the information recording application 41.

[0251] S206: When the terminal communication unit 61 of the meeting device 60 receives a notification to start recording, it assigns a unique recording ID and returns the recording ID to the information recording application 41. The recording ID may be assigned by the information recording application 41 or may be obtained from the information processing system 50.

[0252] S207: The voice acquisition unit 15 of the information recording application 41 acquires the voice data output by the terminal device 10 (voice data received by the remote conference application 42).

[0253] S208: The device communication unit 16 transmits the voice data acquired by the voice acquisition unit 15 and a synthesis request to the meeting device 60.

[0254] S209: The terminal communication unit 61 of the meeting device 60 receives the voice data and the synthesis request, and the voice synthesis unit 65 synthesizes the surrounding voice data collected by the sound collection unit 64 with the received voice data. For example, the voice synthesis unit 65 adds the two pieces of voice data together. Since clear voices around the meeting device 60 are recorded, the accuracy of converting voices around the meeting device 60 (conference room side) into text is improved.

[0255] This voice synthesis can also be performed by the terminal device 10. The recording function may be distributed to the meeting device 60, and the voice processing may be distributed to the terminal device 10. In this case, the load on the meeting device 60 is reduced.

[0256] S210: Furthermore, the panoramic image creation unit 62 of the meeting device 60 creates a panoramic image, and the speaker image creation unit 63 creates a speaker image. S211: The device communication unit 16 of the information recording application 41 repeatedly acquires the panoramic image and the speaker image from the meeting device 60. Furthermore, the device communication unit 16 repeatedly acquires the synthesized voice data from the meeting device 60. These acquisitions may be performed by the device communication unit 16 making a request to the meeting device 60. Alternatively, the meeting device 60 that has received a notice that the camera toggle button 211 is on may automatically transmit the panoramic image and the speaker image. The meeting device 60 that has received a request to synthesize voice data may automatically transmit the synthesized voice data to the information recording application 41.

[0257] S212: The recording control unit 17 of the information recording application 41 creates a combined image by arranging the application screen, the panoramic image, and the speaker image acquired from the remote conference application 42 side by side. The recording control unit 17 repeatedly creates combined images and creates a combined image video by specifying each combined image as a frame that constitutes the video. The recording control unit 17 also saves the audio data received from the meeting device 60.

[0258] The information recording application 41 repeats the above steps S207 to S212.

[0259] S213: When the remote conference is over and recording is no longer necessary, the user instructs the information recording application 41 to end recording (for example, by pressing the recording end button 227). The operation acceptance unit 12 of the information recording application 41 accepts the instruction.

[0260] S214: The device communication unit 16 of the information recording application 41 notifies the meeting device 60 of the end of recording. The meeting device 60 continues creating the panoramic image and the speaker image and synthesizing the voice. However, the meeting device 60 may change the processing load, such as changing the resolution or fps, depending on whether recording is in progress.

[0261] S215: The recording control unit 17 of the information recording application 41 combines the audio data with the combined image moving image to create a combined image moving image with audio.

[0262] S216: Furthermore, if the user checks the checkbox 215 associated with "Automatically transcribe recording after uploading" on the video recording setting screen 210, the audio data processing unit 18 requests the information processing system 50 to convert the audio data into text data. In detail, the audio data processing unit 18 specifies the URL of the save destination via the communication unit 11, and transmits a conversion request for the audio data combined with the combined image video to the information processing system 50 together with the conference ID and video recording ID.

[0263] S217: The communication unit 51 of the information processing system 50 receives the request to convert the voice data, and the text conversion unit 56 converts the voice data into text data using the voice recognition service system 80. The communication unit 51 saves the text data in the same storage destination (URL of the storage service system 70) as the storage destination of the combined image video. In the video recording information storage unit 5002, the text data is associated with the combined image video by the conference ID and the video recording ID. The text data may be managed by the communication management unit 54 of the information processing system 50 and stored in the storage unit 5000. In addition, the terminal device 10 may request voice recognition from the voice recognition service system 80, and the text data acquired from the voice recognition service system 80 may be saved in the storage destination. In addition, the voice recognition service system 80 returns the converted text data to the information processing system 50, but may also directly send it to the URL of the storage destination. The voice recognition service system 80 may select or switch from a plurality of services according to the setting information set by the user in the information processing system 50.

[0264] S218: Furthermore, the upload unit 20 of the information recording application 41 saves the combined image moving image in the storage destination of the combined image moving image via the communication unit 11. In the recording information storage unit 5002, the combined image moving image is associated with the conference ID and the recording ID. The combined image moving image is recorded as having been uploaded.

[0265] S219: The user inputs the end of the meeting to the electronic whiteboard 2. The user may input the end of the meeting to the terminal device 10, and the end of the meeting may be transmitted from the terminal device 10 to the electronic whiteboard 2. In this case, the end of the meeting may be transmitted to the electronic whiteboard 2 via the information processing system 50.

[0266] S220: The communication unit 35 of the electronic whiteboard 2 specifies the conference ID and transmits object data displayed (for example, handwritten) during the conference to the information processing system 50. The communication unit 35 may transmit device identification information of the electronic whiteboard 2 to the information processing system 50. In this case, the conference ID is identified by the association information.

[0267] S221: The information processing system 50 stores the object data in the same storage location as the combined image video and the like based on the conference ID.

[0268] The save destination is notified to the user, so the user can share the combined image video with participants by informing them of the save destination by email, etc. Even if the combined image video, audio data, text data, and object data are created using different devices, they can be collected and stored in one storage location and can be easily viewed by users, etc. later.

[0269] The processing of steps S207 to S212 does not have to be performed in the order shown in FIG. 29, and the synthesis of the audio data and the creation of the combined image may be performed in the reverse order.

[0270] <Major Effects> The meeting device 60 of this embodiment can reduce echo by adjusting the noise reduction level according to the amount of echo components. Since the device is configured to suppress echo by the noise reduction level rather than the gain, there is no fluctuation in the gain (volume) and frequency characteristics (tone) of the sound heard by the user, and therefore it is possible to provide a sound that is easy to hear. In addition, since an echo signal that is difficult for humans to hear is transmitted, it is possible to measure the amount of echo components and suppress the echo without disrupting the conversation.

[0271] <Other application examples> The above describes the best mode for carrying out the present invention using examples, but the present invention is not limited to these examples in any way, and various modifications and substitutions can be made within the scope that does not deviate from the gist of the present invention.

[0272] For example, the conference device of the present invention may be an electronic whiteboard 2 equipped with a microphone, a speaker, an imaging means, and a display control means, a terminal device 10, or other devices for remote conferences, or may be a remote conference app or an information recording app in the device, in addition to the meeting device 60. That is, the present invention may be installed or added as a noise reduction software (program) to various conference devices (devices) or remote conference apps (software). The speaker, microphone, and noise reduction functions may be provided by different devices. For example, the terminal device 10 and the meeting device 60 may be integrated. The meeting device 60 may be externally attached to the terminal device 10. The meeting device 60 may be a celestial sphere camera, a microphone, and a speaker connected by cables.

[0273] The layout of the panoramic image 203, the speaker image 204, and the application screen in the combined image video used in this embodiment is merely an example. The panoramic image 203 may be on the bottom and the speaker image 204 on the top, or the user may change the layout or individually switch between displaying and hiding the panoramic image 203 and the speaker image 204 during playback.

[0274] In addition, the configuration examples in Fig. 9 and the like are divided according to main functions in order to facilitate understanding of the processing by the terminal device 10, the meeting device 60, and the information processing system 50. The present invention is not limited by the manner in which the processing units are divided or the names of the processing units. The processing by the terminal device 10, the meeting device 60, and the information processing system 50 can be divided into even more processing units depending on the processing content. Also, it is possible to divide the processing units so that one processing unit includes even more processes.

[0275] Additionally, the devices described in the examples are merely one of several computing environments for implementing the embodiments disclosed herein. In one embodiment, the information processing system 50 includes a plurality of computing devices, such as a server cluster. The computing devices are configured to communicate with each other via any type of communication link, including a network, shared memory, etc., and perform the processes disclosed herein.

[0276] Furthermore, the information processing system 50 can be configured to share the disclosed processing steps, such as those shown in FIG. 29, in various combinations. For example, a process executed by a specific unit can be executed by multiple information processing devices included in the information processing system 50. Also, the information processing system 50 may be integrated into one server device, or may be divided into multiple devices.

[0277] Each function of the above-described embodiments can be realized by one or more processing circuits. Here, the term "processing circuit" in this specification includes a processor programmed to execute each function by software, such as a processor implemented by an electronic circuit, an ASIC (Application Specific Integrated Circuit), a DSP (Digital Signal Processor), an FPGA (Field Programmable Gate Array), and a device such as a conventional circuit module designed to execute each function described above.

[0278] <Additional Notes> [Claim 1] A conference device having a microphone, a speaker, and a noise reduction function, an echo signal output unit that outputs an echo signal from the speaker; an echo component detection unit that detects an echo component from the echo signal output from the speaker and input to the microphone; a control unit that controls a level of noise reduction by the noise reduction function in accordance with the magnitude of the echo component amount detected by the echo component detection unit; A conference device having the above configuration. [Claim 2] 2. The conference device according to claim 1, wherein the echo signal is an ultrasonic wave in a frequency band of 16,000 Hz or more. [Claim 3] 3. The conference device according to claim 1, wherein the echo signal output unit outputs the echo signal when a sound input to the microphone and an sound output from the speaker are in a silent interval. [Claim 4] 4. The conference device according to claim 1, wherein the noise reduction function allows adjustment of a noise reduction level in at least two stages. [Claim 5] 5. The conference device according to claim 4, wherein the control unit reduces a level of the noise reduction when the magnitude of the echo component amount detected by the echo component detection unit is zero or can be regarded as zero. [Claim 6] 5. The conference device according to claim 4, wherein the control unit increases a level of the noise reduction when the magnitude of the echo component amount detected by the echo component detection unit is greater than a threshold value. [Claim 7] 5. The conference device according to claim 4, wherein the control unit does not change the level of the noise reduction when the magnitude of the echo component amount detected by the echo component detection unit is greater than zero and is not greater than a threshold value. [Claim 8] Each time the echo signal output unit detects the silent section and outputs the echo signal, the control unit reduces a level of the noise reduction when the magnitude of the echo component amount detected by the echo component detection unit is zero or can be regarded as zero; 4. The conference device according to claim 3, wherein, when the magnitude of the echo component amount detected by the echo component detection section is greater than a threshold value, the process of increasing the noise reduction level is repeated. [Claim 9] Communicate with a second conference device at another location; the echo signal output unit of the second conference device outputs the echo signal from the speaker of the second conference device; Transmitting the echo signal acquired by the microphone of the second conference device to the second conference device; the echo component detection unit of the conference device detects the echo component from the echo signal output from a speaker of the conference device and acquired by a microphone of the conference device; 9. The conference device according to claim 1, wherein the control unit of the conference device controls a level of noise reduction by a noise reduction function of the conference device in accordance with the magnitude of the echo component amount. [Claim 10] The conference device is a meeting device equipped with a microphone, a speaker, and an imaging means capable of imaging a 360° surrounding area, The meeting device includes: Detecting an echo component from the echo signal output by the speaker of the meeting device and input to the microphone of the meeting device; the meeting device controls a level of noise reduction by the noise reduction function in accordance with the magnitude of the echo component amount detected by the echo component detection unit; The conferencing device of claim 1. [Claim 11] the conference device is an electronic whiteboard equipped with a microphone, a speaker, an imaging means, and a display control means; The electronic whiteboard detects an echo component from the echo signal output from the speaker of the electronic whiteboard and input to the microphone of the electronic whiteboard, the electronic whiteboard controls a level of noise reduction by the noise reduction function in accordance with the magnitude of the echo component amount detected by the echo component detection unit. The conferencing device of claim 1. [Explanation of symbols]

[0279] 10 Terminal Equipment 50 Information Processing Systems 60 Meeting Devices 100 Recording Information Creation System [Prior art documents] [Patent documents]

[0280] [Patent Document 1] JP 2008-312002 A

Claims

1. A conference device having a microphone, a speaker, and a noise reduction function, an echo signal output unit that outputs an echo signal from the speaker; an echo component detection unit that detects an echo component from the echo signal output from the speaker and input to the microphone; a control unit that controls a level of noise reduction by the noise reduction function in accordance with the magnitude of the echo component amount detected by the echo component detection unit; A conference device having the above configuration.

2. 2. The conference device according to claim 1, wherein the echo signal is an ultrasonic wave having a frequency band of 16,000 Hz or more.

3. 3. The conference device according to claim 1, wherein the echo signal output unit outputs the echo signal when a sound input to the microphone and a sound output from the speaker are in a silent section.

4. 2. The conference device according to claim 1, wherein the noise reduction function allows the noise reduction level to be adjusted in at least two stages.

5. 5. The conference device according to claim 4, wherein the control unit reduces a level of the noise reduction when the magnitude of the echo component amount detected by the echo component detection unit is zero or can be regarded as zero.

6. 5. The conference device according to claim 4, wherein the control unit increases a level of the noise reduction when the magnitude of the echo component amount detected by the echo component detection unit is greater than a threshold value.

7. 5. The conference device according to claim 4, wherein the control unit does not change the level of the noise reduction when the magnitude of the echo component amount detected by the echo component detection unit is greater than zero and is not greater than a threshold value.

8. Each time the echo signal output unit detects the silent section and outputs the echo signal, the control unit reduces a level of the noise reduction when the magnitude of the echo component amount detected by the echo component detection unit is zero or can be regarded as zero; 4. The conference device according to claim 3, wherein when the magnitude of the echo component amount detected by the echo component detection section is greater than a threshold value, the process of increasing the noise reduction level is repeated.

9. Communicate with a second conference device at another location; the echo signal output unit of the second conference device outputs the echo signal from the speaker of the second conference device; Transmitting the echo signal acquired by the microphone of the second conference device to the second conference device; the echo component detection unit of the conference device detects the echo component from the echo signal output from a speaker of the conference device and acquired by a microphone of the conference device; 2. The conference device according to claim 1, wherein the control unit of the conference device controls a level of noise reduction by a noise reduction function of the conference device in accordance with the magnitude of the echo component amount.

10. The conference device is a meeting device equipped with a microphone, a speaker, and an imaging means capable of imaging a 360° surrounding area, The meeting device includes: Detecting an echo component from the echo signal output by the speaker of the meeting device and input to the microphone of the meeting device; the meeting device controls a level of noise reduction by the noise reduction function in accordance with the magnitude of the echo component amount detected by the echo component detection unit; The conference device of claim 1.

11. the conference device is an electronic whiteboard equipped with a microphone, a speaker, an imaging means, and a display control means; The electronic whiteboard detects an echo component from the echo signal output from the speaker of the electronic whiteboard and input to the microphone of the electronic whiteboard, the electronic whiteboard controls a level of noise reduction by the noise reduction function in accordance with the magnitude of the echo component amount detected by the echo component detection unit. The conference device of claim 1.

12. An equipment system in which a first conference device and a second conference device having a microphone, a speaker, and a noise reduction function can communicate with each other, The first conference device an echo signal output unit that outputs an echo signal from a speaker of the first conference device; a first echo signal acquisition unit that acquires the echo signal output from a speaker of the first conference device and input to a microphone of the first conference device, The second conference device outputting the echo signal received from the first conference device from a speaker of the second conference device; a second echo signal acquisition unit that acquires the echo signal input to a microphone of the second conference device; an echo component detection unit that detects an echo component from the echo signal acquired by the second echo signal acquisition unit; a control unit that controls a level of noise reduction by the noise reduction function in accordance with the magnitude of the echo component amount detected by the echo component detection unit; An equipment system having the above configuration.

13. An echo suppression method performed by a conference device having a microphone, a speaker, and a noise reduction function, comprising: outputting an echo signal from the speaker; A process of detecting an echo component from the echo signal output from the speaker and input to the microphone; A process of controlling a level of noise reduction by the noise reduction function according to the magnitude of the detected echo component amount; An echo suppression method.

14. A conference device having a microphone, a speaker, and a noise reduction function, an echo signal output unit that outputs an echo signal from the speaker; an echo component detection unit that detects an echo component from the echo signal output from the speaker and input to the microphone; a control unit that controls a level of noise reduction by the noise reduction function in accordance with the magnitude of the echo component amount detected by the echo component detection unit; A program to function as a

Images