Speech processing systems, devices, and methods

Abstract

To provide a speech processing system that lightens a processing load of conversion to text data.SOLUTION: The present invention relates to a speech processing system in which a terminal unit and a device comprising a microphone communicate with each other, and the speech processing system has: a speech synthesis part which synthesizes a first speech that the terminal unit receives from outside and a second speech that the device collects to generate a synthetic speech; and a text conversion request part which sends a request to convert the synthetic speech that the speech synthesis part generates into text data to the outside.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present invention relates to an audio processing system, a device, and an audio processing method.

Background Art

[0002] There is known a remote conferencing service system that transmits images and sounds in real time from one base to one or more other bases and enables users at remote locations to hold meetings using images and sounds. Also, as a method of recording the content of remote communication such as a meeting as minutes, there is known a method of converting the voices of users at each base into text and storing the text data.

[0003] There is known a technique for associating with text data which speaker among a plurality of speakers made a speech (see, for example, Patent Document 1). Patent Document 1 discloses an information processing apparatus that discriminates a speaker who is making a speech by discriminating the voiceprint of the speaker by voiceprint identification and adds identification information of any speaker to each speech sentence.

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, the conventional technology has a problem that the processing load for conversion into text data is large. That is, since the conventional technology separately performs speech recognition on each speech made by a plurality of speakers, there is a risk of straining resources for speech recognition and a risk of taking time for recognition.

[0005] In view of the above problems, an object of the present invention is to provide a technique for reducing the processing load for conversion into text data.

Means for Solving the Problems

[0006] In view of the above problems, the present invention is an audio processing system in which a terminal device and a device provided with a microphone communicate with each other, The terminal device receives the first voice from a terminal device at another location via the network.A speech synthesis unit that generates synthesized speech by combining a first voice received from an external source by the terminal device and a second voice collected by the device, and a text conversion request unit that requests an external source to convert the synthesized speech synthesized by the speech synthesis unit into text data. Before the speech synthesis unit synthesizes the first voice and the second voice, a location determination processing unit determines the location from which the first voice and the second voice were spoken based on the sound pressure of the first voice and the second voice, It is characterized by having the following features. [Effects of the Invention]

[0007] This system can provide a speech processing system that reduces the processing load of converting speech data to text data. [Brief explanation of the drawing]

[0008] [Figure 1] This diagram outlines the creation of recording information that saves the screen of an application running during a remote meeting, along with a panoramic image of the surroundings. [Figure 2] This is an example of a system block diagram relating to the functions of a record information creation system. [Figure 3] This diagram shows an example configuration of a record information creation system. [Figure 4] This figure shows the hardware configuration of an example of an information processing system and terminal device. [Figure 5] This is an example of a hardware configuration diagram for a meeting device capable of capturing 360° video. [Figure 6] This diagram illustrates the imaging range of the meeting device. [Figure 7] This diagram illustrates the process of extracting images from a panoramic image and a speaker image. [Figure 8] This is an example of a functional block diagram that explains the functions of terminal devices, meeting devices, and information processing systems in a record information creation system by dividing them into blocks. [Figure 9] This figure shows an example of video recording information stored in the information storage unit. [Figure 10] This figure shows an example of meeting information stored in the meeting information storage unit, which is managed by the meeting information acquisition unit. [Figure 11]It is a diagram showing an example of recording information stored in the recording information storage unit. [Figure 12] It is a diagram explaining the structure of text data stored in the storage service system. [Figure 13] It is a diagram showing an example of the initial screen after login displayed by the information recording application operating on the terminal device. [Figure 14] It is a diagram showing an example of the recording setting screen displayed by the information recording application. [Figure 15] It is a diagram showing an example of the meeting list screen displayed by the information recording application. [Figure 16] It is a diagram showing an example of the meeting record confirmation screen presented by the meeting record confirmation unit and displayed on the terminal device. [Figure 17] It is a diagram explaining an example of the base determination conditions. [Figure 18] It is an example of a flowchart diagram explaining the method of determining the base according to the base determination condition 1-1. [Figure 19] It is an example of a flowchart diagram explaining the method of determining the base according to the base determination condition 1-2. [Figure 20] It is an example of a flowchart diagram explaining the method of determining the base according to the base determination condition 2. [Figure 21] It is a diagram explaining the method for the user to associate the base identification information E, the speech recognition result D, and the synthesized speech C in the queue method. [Figure 22] It is a diagram explaining the method for the user to associate the base identification information E, the speech recognition result D, and the synthesized speech C in the time stamp method. [Figure 23] It is an example of a sequence diagram showing the procedure for the information recording application to record the panoramic image, the speaker image, and the application screen. ] [Figure 24] It is an example of a sequence diagram showing the procedure for the information recording application to record the panoramic image, the speaker image, and the application screen. [Figure 25] It is an example of a sequence diagram explaining the process for the user to edit the base identification information E.

Embodiments for Carrying Out the Invention

[0009] Hereinafter, as an example of an embodiment for implementing the present invention, a recording information creation system and a voice processing method performed by the recording information creation system will be described.

[0010] <An example of a method for creating text data in a remote conference> First, referring to FIG. 1, an outline of a method for creating minutes using a panoramic image and an app screen will be described. FIG. 1 is a diagram for explaining an outline of the creation of recording information for saving the screen of an app executed during a remote conference together with a surrounding panoramic image. As shown in FIG. 1, a user at the home base 102 shown is conducting a remote conference with another base 101 using the remote conference service system 90.

[0011] The recording information creation system 100 of the present embodiment creates recording information (minutes) using a horizontal panoramic image (hereinafter referred to as a panoramic image) captured by a meeting device 60 equipped with a microphone and a speaker, and a screen created by an application (hereinafter referred to as an app) executed by the terminal device 10. Regarding voice, the recording information creation system 100 (an example of a voice processing system) synthesizes the voice received by the remote conference app 42 and the voice acquired by the meeting device 60 and includes it in the recording information. Hereinafter, an outline will be described.

[0012] (1) In the terminal device 10, an information recording app 41 and a remote conference app described later are operating. In addition, an app for displaying materials or the like may also be operating. The information recording app 41 transmits the voice output from the terminal device 10 (including the voice received from another base by the remote conference app. An example of first voice data.) to the meeting device 60. The meeting device 60 mixes (synthesizes) the voice it has acquired (an example of second voice data) and the voice of the remote conference app. The meeting device 60 determines the base where the speech was made based on the sound pressure of the voice output from the terminal device 10.

[0013] (2) The meeting device 60 is equipped with a microphone and, based on the direction from which the sound was acquired, processes the image to extract the speaker from the panoramic image and creates a speaker image. The meeting device 60 transmits both the panoramic image and the speaker image to the terminal device 10.

[0014] (3) The information recording application 41 running on the terminal device 10 can display the panoramic image 203 and the speaker image 204. The information recording application 41 combines the screen of any application selected by the user (for example, the screen 103 of the remote conferencing application) with the panoramic image 203 and the speaker image 204. For example, the panoramic image 203, the speaker image 204, and the application screen 103 are combined so that the panoramic image 203 and the speaker image 204 are on the left and the screen 103 of the remote conferencing application is on the right (hereinafter referred to as the combined image 105). Since the process in (3) is executed repeatedly, the combined image 105 becomes a video (hereinafter referred to as the combined image video). The information recording application 41 also combines synthesized audio (hereinafter referred to as synthesized audio) with the combined image video to create a video with audio.

[0015] In this embodiment, an example is described in which the panoramic image 203, speaker image 204, and application screen 103 are combined. However, the information recording application 41 may save these separately and arrange them on the screen during playback.

[0016] (4) The information recording application 41 accepts editing work (such as cutting out unnecessary parts by the user) and completes the combined image video. The combined image video constitutes part of the recorded information.

[0017] (5) The information recording application 41 sends the created combined image video (with audio) to the storage service system 70 for storage.

[0018] (6) In the case of batch conversion of audio, the information recording application 41 extracts only the audio from the combined image video (it may also keep the audio before combining) and sends the extracted audio to the information processing system 50. The information processing system 50 sends the audio to the speech recognition service system 80, which converts the audio into text data. The text data also includes data on how many minutes after the start of recording the speaker spoke.

[0019] (7) In the case of real-time conversion, the meeting device 60 transmits the synthesized speech C to the information processing system 50 after determining the location. The information processing system 50 converts it into text data in real time using the speech recognition service system and transmits this text data to the information recording application 41. This embodiment mainly describes the case of real-time conversion.

[0020] Furthermore, the information processing system 50 can perform billing processing for users based on the services they have used. For example, billing is calculated based on the amount of text data, the file size of combined images and videos, processing time, etc.

[0021] (8) The information processing system 50 additionally stores text data in the storage service system 70 which stores the combined image video. Users can play the combined image video on the terminal device 10. The text data constitutes part of the recorded information.

[0022] Thus, the combined video image displays and records a panoramic image of the surroundings including the user, an image of the speaker, and the screen of the application displayed during the remote meeting. Since speech recognition is performed on the synthesized speech, the processing load on the speech recognition service system can be reduced compared to performing speech recognition separately. In addition, since the location from which the synthesized speech was spoken is determined based on sound pressure information, it is possible to record which location the speech data was spoken at.

[0023] <About Terminology> An application (or app) is software developed and used for a specific function or purpose, and is not intended for the operation of a computer itself. Applications include native apps and web apps.

[0024] A running app refers to an app in its state from the time it is launched until it is closed. The app does not need to be active (the app in the foreground); it only needs to be running in the background.

[0025] A device is a device that can capture images of its surroundings and collect sounds from its surroundings. In this embodiment, it will be referred to as a meeting device 60.

[0026] The surrounding image acquired by the meeting device 60 refers to an image captured with a wider field of view horizontally than the normal field of view. In this embodiment, the surrounding image is described using the term panoramic image. A panoramic image is generally an image with a field of view of approximately 180° to 360° horizontally. The meeting device 60 does not necessarily have to capture a panoramic image with a single unit; it may be a combination of multiple imaging devices with normal fields of view.

[0027] Recorded information refers to the information recorded by the information recording application 41. If the information recording application 41 records the screen of a remote conferencing application, the recorded information may become meeting minutes. Recorded information includes, for example, combined video images (including audio) and text data from which the audio has been recognized by speech recognition.

[0028] A tenant is a group of users (companies, local governments, and some of these organizations) that have contracted to receive services from a service provider. The creation of record information and its conversion to text data in this embodiment are performed because the tenant has a contract with the service provider.

[0029] Remote communication refers to communication with someone located at a physically distant location, using software and terminal devices to communicate via voice and video. An example of remote communication is a remote conference, which can also be called a meeting, discussion, gathering, or assembly, while a seminar can be called a training session, study group, workshop, or workshop.

[0030] A base of operations refers to a place that serves as the foundation for activities. An example of a base of operations is a conference room. A conference room is a room set up primarily for the purpose of holding meetings.

[0031] Speech refers to human speech sounds and ambient sounds, while speech data is the data representation of speech. However, in this embodiment, we will not strictly distinguish between the two.

[0032] <System block diagram related to functions> Figure 2 is a system block diagram relating to the functions of the recording information creation system 100. The recording information creation system 100 has a local audio input unit 7 that acquires local audio A during remote conferences, and a third-party audio input unit 8 that acquires audio B from other locations.

[0033] The local audio input unit 7 can be, for example, a general-purpose microphone. The other-site audio input unit 8 is a module that can acquire audio B from another site, separate from the local audio input unit 7. The other-site audio input unit 8 could be, for example, a USB connector, Bluetooth®, Wi-Fi, or other wireless communication module that can acquire audio B from another site via a connection unit that connects the terminal device participating in the remote conference to the recording information creation system 100.

[0034] With the above configuration, the recording information creation system 100 can acquire audio from its own location A and audio from another location B as separate audio sources, even in remote conferences using a general remote communication system, without requiring multiple dedicated terminals or accounts.

[0035] Furthermore, the local voice A and the other-site voice B are input to the speech synthesis unit 65, and after synthesis processing, they are output as synthesized voice C. Synthesized voice C is converted into text by the speech recognition unit 55 using speech recognition technology and stored as speech recognition result D in the speech recognition result recording unit 57. Furthermore, synthesized voice C is recorded as audio data by the audio recording unit 56.

[0036] As a speech recognition technology, the speech recognition service system 80 can utilize a general-purpose speech recognition engine. The implementation of the speech recognition engine can be selected for each system, including embedding it in hardware or using it as a cloud service.

[0037] By combining the local voice A and the voice B from another location into synthesized voice C, the amount of data does not increase significantly even in environments with poor communication conditions, allowing voice data to be input to the voice recognition unit 55.

[0038] Furthermore, if speech recognition is performed without synthesizing the audio from multiple locations, it becomes necessary to process two speech recognitions in parallel, which raises concerns about performance degradation during speech recognition and may impair real-time capabilities. However, by synthesizing the audio from multiple locations and inputting the synthesized speech C as audio data to the speech recognition unit 55, it becomes possible to convert speech to text with low performance and low cost, and real-time capabilities are also improved. Moreover, by performing speech recognition on synthesized speech C, it becomes possible to perform speech recognition that estimates context, improving the speech recognition rate.

[0039] Furthermore, because the audio recording unit 56 records the synthesized speech C, neither of the pre-synthesis audio data is lost, nor is the timing of speech shifted during playback. As a result, the simultaneously spoken audio is recorded as is, allowing users to review the meeting audio later as a meeting record without any sense of incongruity.

[0040] The recording information creation system 100 may also have a voice data processing unit that processes and modifies voice data before voice recognition or voice synthesis.

[0041] Furthermore, the recording information creation system 100 has a base determination processing unit 64. The base determination processing unit 64 has a sound pressure detection unit 64a that detects local base sound pressure information A' and other base sound pressure information B' from local base voice A and other base voice B, respectively. The base determination unit 64b generates base identification information E by determining whether the synthesized voice C is mainly composed of local base voice A or other base voice B by comparing local base sound pressure information A' and other base sound pressure information B'. The base identification information E is stored in the base determination result recording unit 58.

[0042] By using a simple method of comparing sound pressure information to identify the source of a speech, it becomes possible to identify the source without using speaker identification AI such as voice recognition AI that uses voiceprints or facial recognition AI that uses the speaker's facial image. This allows for low-performance source identification and low-cost implementation.

[0043] Furthermore, the recording information creation system 100 includes a meeting record confirmation unit 59 that synchronously reads out the location identification information E, the speech recognition result D, and the synthesized speech C from the location determination result recording unit 58, the speech recognition result recording unit 57, and the speech recording unit 56, respectively, and displays them to the user.

[0044] As a result, users will be able to determine whether the speech recognition results are based on speech from their own location or from another location, thereby improving their understanding of meeting minutes.

[0045] <Example System Configuration> Next, the system configuration of the recording information creation system 100 will be explained with reference to Figure 3. Figure 3 shows an example of the configuration of the recording information creation system 100. In Figure 3, one of several locations where remote meetings are held (local location 102) is shown, and the terminal device 10 at local location 102 communicates with the information processing system 50, the storage service system 70, and the remote meeting service system 90 via the network. A meeting device 60 is also located at local location 102, and the terminal device 10 is connected to this meeting device 60 via a USB cable or the like for communication.

[0046] Terminal device 10 operates at least an information recording application 41 and a remote conferencing application 42. The remote conferencing application 42 can communicate with terminal devices 10 at other locations 101 via a remote conferencing service system 90 on the network, allowing users at each location to hold meetings remotely. The information recording application 41 creates recording information for remote meetings conducted by the remote conferencing application 42 using the functions of the information processing system 50 and the meeting device 60.

[0047] In this embodiment, we describe an example of creating recorded information during a remote meeting, but the meeting to be recorded does not have to be a meeting that communicates with remote locations. In other words, the meeting may be one in which only participants from within a single location are present. In this case, only the audio collected by the meeting device 60 is saved without synthesis, and there is no change in the processing of the information recording application 41.

[0048] The terminal device 10 has a built-in (or external) camera with a normal field of view that captures an image of the front, including the user 107 operating the terminal device 10. A normal field of view is an image that is not a panoramic image, but in this embodiment, it is mainly a planar image that is not curved, such as a 360-degree image. The terminal device 10 also has a built-in (or external) microphone that collects ambient sounds from the user operating the terminal device 10. Therefore, the user can conduct a conventional remote conference using the remote conference application 42 without being aware of the information recording application 41. The information recording application 41 and the meeting device 60 do not affect the remote conference application 42, except for an increase in the processing load of the terminal device 10. The remote conference application 42 can also transmit the panoramic image and speaker image captured by the meeting device 60 to the remote conference service system 90.

[0049] The information recording application 41 communicates with the meeting device 60 to create recorded information. The meeting device 60 is a meeting device equipped with a panoramic image capture device, a microphone, and a speaker. While the camera on the terminal device 10 can only capture a limited area in front of it, the meeting device 60 can capture the entire surrounding area (not necessarily all around). The meeting device 60 can always keep multiple participants 106, as shown in Figure 3, within its field of view.

[0050] In addition, the meeting device 60 extracts speaker images from the panoramic image, and combines the audio acquired by the meeting device 60 with the audio output by the terminal device 10 (including the audio received by the remote conferencing application 42). The meeting device 60 is not limited to being placed on a desk; it can be placed anywhere on the site 102. Since the meeting device 60 can capture a 360-degree image, it can be placed on the ceiling, for example.

[0051] The information recording application 41 displays a list of applications running on the terminal device 10, performs image synthesis (creation of combined image videos) for the recorded information described above, plays the combined image videos, accepts editing requests, etc. The information recording application 41 also displays a list of remote meetings that have been held or are scheduled to be held, etc. The list of remote meetings is used for information related to the recorded information, allowing the user to link remote meetings with the recorded information.

[0052] The remote conferencing application 42 handles communication connections with other locations 101, sending and receiving images and audio with other locations 101, displaying images, outputting audio, etc.

[0053] The information recording application 41 and the remote conferencing application 42 may be either web applications or native applications. A web application is an application in which a program on a web server and a program on a web browser work together to perform processing, and does not require installation on the terminal device 10. A native application is an application that is installed on the terminal device 10 and used thereafter. In this embodiment, both will be described as native applications.

[0054] The terminal device 10 may be a general-purpose information processing device equipped with communication functions, such as a PC (Personal Computer), smartphone, or tablet device. Other examples of terminal devices 10 include electronic whiteboards, game consoles, PDAs (Personal Digital Assistants), wearable PCs, car navigation systems, industrial machinery, medical equipment, and network-connected home appliances. The terminal device 10 only needs to be a device on which the information recording application 41 and the remote conferencing application 42 run.

[0055] The information processing system 50 consists of one or more information processing devices located on a network. The information processing system 50 has one or more server applications that perform processing in cooperation with the information recording application 41, and infrastructure services. These server applications retrieve a list of remote meetings managed by the meeting management system 9, manage recorded information from remote meetings, and manage various settings and storage paths. The infrastructure services handle user authentication, contracts, billing, etc.

[0056] The meeting management system 9 is a system that handles tasks such as reserving meeting rooms and managing meeting schedules. The meeting management system 9 and the information processing system 50 may be integrated into a single system.

[0057] Furthermore, all or part of the functions of the information processing system 50 may reside in a cloud environment or in an on-premises environment. The information processing system 50 may consist of multiple server devices or a single information processing device. For example, server applications and infrastructure services may be provided by separate information processing devices, and furthermore, there may be separate information processing devices for each function within the server application. The information processing system 50 may be integrated with the storage service system 70 and the speech recognition service system 80 described below.

[0058] The storage service system 70 is a storage means on a network and provides a storage service that accepts the storage of files, etc. Examples of storage service systems 70 include OneDrive®, Google Workspace®, and Dropbox®. The storage service system 70 may also be an on-premises NAS (Network Attached Storage).

[0059] The speech recognition service system 80 provides a service that performs speech recognition on speech data and converts it into text data. The speech recognition service system 80 may be a general-purpose commercial service or it may be part of the functions of the information processing system 50.

[0060] <Example Hardware Configuration> Referring to Figure 4, the hardware configuration of the information processing system 50 and terminal device 10 according to this embodiment will be described.

[0061] <<Information Processing Systems and Terminal Devices>> Figure 4 shows the hardware configuration of an example of the information processing system 50 and terminal device 10 according to this embodiment. As shown in Figure 4, the information processing system 50 and terminal device 10 are built using a computer and include a CPU 501, ROM 502, RAM 503, HD (Hard Disk) 504, HDD (Hard Disk Drive) controller 505, display 506, external device connection I / F (Interface) 508, network I / F 509, bus line 510, keyboard 511, pointing device 512, optical drive 514, and media I / F 516.

[0062] Of these, the CPU 501 controls the operation of the entire information processing system 50 and terminal device 10. The ROM 502 stores programs used to drive the CPU 501, such as IPL. The RAM 503 is used as the work area for the CPU 501. The HD 504 stores various data such as programs. The HDD controller 505 controls the reading or writing of various data to the HD 504 according to the control of the CPU 501. The display 506 displays various information such as cursors, menus, windows, characters, or images. The external device connection I / F 508 is an interface for connecting various external devices. In this case, external devices include, for example, USB (Universal Serial Bus) memory and printers. The network I / F 509 is an interface for data communication using the network N2. The bus line 510 is an address bus, data bus, etc., for electrically connecting each component such as the CPU 501 shown in Figure 4.

[0063] The keyboard 511 is a type of input means equipped with multiple keys used for inputting characters, numbers, or various instructions. The pointing device 512 is a type of input means used for selecting and executing various instructions, selecting processing targets, moving the cursor, etc. The optical drive 514 controls the reading or writing of various data to the optical storage medium 513, which is an example of a removable recording medium. The optical storage medium 513 may be a CD, DVD, Blu-ray (registered trademark), etc. The media I / F 516 controls the reading or writing (storage) of data to the recording medium 515, such as flash memory.

[0064] <<Meeting Devices>> The hardware configuration of the meeting device 60 will be explained using Figure 5. Figure 5 is an example of a hardware configuration diagram for a meeting device 60 capable of capturing 360° video. In the following, the meeting device 60 is assumed to be a device that uses an image sensor to capture 360° video around the device at a predetermined height, but it may have one or more image sensors. Furthermore, it does not necessarily have to be a dedicated device; a 360° video capture unit can be attached to a PC, digital camera, smartphone, etc., to achieve substantially the same function.

[0065] As shown in Figure 5, the meeting device 60 consists of an imaging unit 601, an image processing unit 604, an imaging control unit 605, a microphone 608, a sound processing unit 609, a CPU (Central Processing Unit) 611, a ROM (Read Only Memory) 612, an SRAM (Static Random Access Memory) 613, a DRAM (Dynamic Random Access Memory) 614, an operation unit 615, an external device connection interface 616, a communication unit 617, an antenna 617a, a sound sensor 618, and a recessed terminal 621 for Micro USB.

[0066] Of these, the imaging unit 601 includes a wide-angle lens (a so-called fisheye lens) 602 with a 360° field of view for forming a hemispherical image, and an image sensor 603 provided in conjunction with each wide-angle lens. The image sensor 603 includes an image sensor such as a CMOS (Complementary Metal Oxide Semiconductor) sensor or a CCD (Charge Coupled Device) sensor that converts the optical image from the fisheye lens 602 into electrical signal image data and outputs it, a timing generation circuit that generates horizontal or vertical synchronization signals and pixel clocks for this image sensor, and a group of registers in which various commands and parameters necessary for the operation of this image sensor are set.

[0067] Each image sensor 603 of the imaging unit 601 is connected to the image processing unit 604 via a parallel I / F bus. On the other hand, the image sensor 603 of the imaging unit 601 is connected to the imaging control unit 605 via a serial I / F bus (such as an I2C bus). The image processing unit 604, the imaging control unit 605, and the sound processing unit 609 are connected to the CPU 611 via bus 610. Furthermore, ROM 612, SRAM 613, DRAM 614, operation unit 615, external device connection I / F 616, communication unit 617, and sound sensor 618 are also connected to bus 610.

[0068] The image processing unit 604 receives image data output from the image sensor 603 via a parallel I / F bus, performs predetermined processing on each image data, and creates panoramic images and speaker images from the fisheye video. Furthermore, the image processing unit 604 combines the panoramic image and speaker images, etc., to output a single video.

[0069] The imaging control unit 605 generally uses the I2C bus to set commands and other information in the registers of the image sensor 603, with the imaging control unit 605 acting as the master device and the image sensor 603 as the slave device. It receives necessary commands and other information from the CPU 611. The imaging control unit 605 also uses the I2C bus to acquire status data and other information from the registers of the image sensor 603 and send it to the CPU 611.

[0070] Furthermore, the imaging control unit 605 instructs the image sensors 603a and 603b to output image data when the image capture start button on the operation unit 615 is pressed or when an image capture start instruction is received from the PC. Depending on the meeting device 60, there may also be functions to display a preview or video on a display (for example, the display of a PC or smartphone). In this case, the image data output from the image sensor 603 is performed continuously at a predetermined frame rate (frames / minute).

[0071] Furthermore, as will be described later, the imaging control unit 605 also functions as a synchronization control means that works in cooperation with the CPU 611 to synchronize the output timing of image data from the image sensor 603. In this embodiment, the meeting device 60 is not provided with a display, but a display unit may be provided.

[0072] Microphone 608 converts sound into sound (signal) data. The sound processing unit 609 receives the audio data output from microphone 608 through the I / F bus and performs predetermined processing on the audio data.

[0073] The CPU 611 controls the overall operation of the meeting device 60 and performs necessary processing. The ROM 612 stores various programs for the CPU 611. The SRAM 613 and DRAM 614 are work memories that store programs executed by the CPU 611 and data in progress. In particular, the DRAM 614 stores image data in progress and processed equirectangular projection image data from the image processing unit 604.

[0074] The control unit 615 is a collective term for the control buttons, such as the image capture start button 615a. By operating the control unit 615, the user can start imaging and recording, turn the power on / off, establish a communication connection, and input various settings such as imaging modes and imaging conditions.

[0075] The external device connection interface 616 is an interface for connecting various external devices. In this case, external devices include, for example, a PC (Personal Computer). Video data and image data stored in DRAM 614 are transmitted to an external terminal or recorded on external media via this external device connection interface 616.

[0076] The communication unit 617 may communicate with a cloud server via the internet using wireless communication technology such as Wi-Fi through the antenna 617a provided on the meeting device 60, and transmit the stored video data and image data to the cloud server. Alternatively, the communication unit 617 may communicate with nearby devices using short-range wireless communication technology such as BLE (Bluetooth Low Energy; registered trademark) or NFC.

[0077] The audio sensor 618 is a sensor that acquires 360° audio data in order to determine from which direction loud audio is being input within a 360° radius (horizontal plane) around the meeting device 60. The audio processing unit 609 identifies the direction of the strongest sound based on the input 360° audio parameters and outputs the audio input direction within the 360°.

[0078] Furthermore, other sensors (such as compass / accelerometers or GPS) can be used to calculate direction, position, angle, acceleration, etc., and these can be used for image correction or to add location information.

[0079] The image processing unit 604 also performs the following processing:

[0080] The CPU 611 creates panoramic images in the following way: The CPU 611 performs predetermined camera image processing, such as Bayer conversion (RGB interpolation), on the RAW data input from the image sensor that inputs spherical images to create a fisheye image (curved image). Furthermore, it performs DeWarp processing (distortion correction processing) on ​​the created fisheye image (curved image) to create a panoramic image (flat image) that captures 360° around the meeting device 60.

[0081] The CPU 611 creates speaker images in the following way: The CPU 611 creates speaker images by extracting the speaker from a panoramic image (flat image) that captures the surrounding 360°. The CPU 611 uses the direction of the audio input, identified from the 360° output by the audio sensor 618 and the audio processing unit 609, as the direction of the speaker, and extracts the speaker image from the panoramic image. In this case, the method for extracting a person's image from the direction of the audio input is to cut out 30° around the audio direction identified from the 360°, and then perform face detection (person detection, etc., is also acceptable) within that area to extract the image. The CPU 611 further identifies speaker images for a specific number of people (e.g., 3 people) who have spoken most recently from the extracted speaker images.

[0082] The panoramic image and the images of one or more speakers may be sent individually to the information recording application 41, or the meeting device 60 may create a single image from them and send it to the information recording application 41. In this embodiment, the panoramic image and the images of one or more speakers are sent individually from the meeting device 60 to the information recording application 41.

[0083] Figure 6 illustrates the imaging range of the meeting device 60. As shown in Figure 6(a), the meeting device 60 images a 360° range in the horizontal direction. As shown in Figure 6(b), the meeting device 60 has an imaging range that is 0° horizontal to the height of the meeting device 60, and a predetermined angle above and below it.

[0084] Figure 7 illustrates the extraction of panoramic and speaker images. As shown in Figure 7, the image captured by the meeting device 60 forms a part 110 of a sphere and therefore has a three-dimensional shape. As shown in Figure 6(b), the meeting device 60 divides the field of view into predetermined angles vertically and horizontally and performs perspective projection transformation. By performing perspective projection transformation seamlessly across the entire 360° horizontal direction, a predetermined number of planar images are obtained, and by stitching these predetermined number of planar images together horizontally, a panoramic image 111 is obtained. The meeting device 60 also performs face detection from the panoramic image within a predetermined range centered on the direction of sound, and creates speaker images 112 by extracting 15° to the left and right (30° in total) from the center of the face.

[0085] <About the features> Next, with reference to Figure 8, the functional configuration of the recording information creation system 100 will be described. Figure 8 is an example of a functional block diagram that explains the functions of the terminal device 10, meeting device 60, and information processing system 50 in the recording information creation system 100 by dividing them into blocks.

[0086] <<Terminal device>> The information recording application 41 operating on the terminal device 10 includes a communication unit 11, an operation reception unit 12, a display control unit 13, an application screen acquisition unit 14, an audio acquisition unit 15, a device communication unit 16, a video storage unit 17, an audio data processing unit 18, a recording playback unit 19, an upload unit 20, an editing processing unit 21, and a search unit 22. Each of these units in the terminal device 10 is a function or means of function realized by any of the components shown in Figure 4 operating according to instructions from the CPU 501 following the information recording application 41 deployed from the HD 504 to the RAM 503. The terminal device 10 also has a storage unit 1000 constructed from the HD 504, etc., as shown in Figure 4. An information storage unit 1001 is constructed in the storage unit 1000.

[0087] The communication unit 11 communicates various types of information with the information processing system 50 via the network. For example, the communication unit 11 receives a list of remote meetings from the information processing system 50 and sends voice data recognition requests to the information processing system 50.

[0088] The display control unit 13 displays various screens that serve as the user interface in the information recording application 41 according to the screen transitions set in the information recording application 41. The operation reception unit 12 receives various operations on the information recording application 41.

[0089] The application screen acquisition unit 14 acquires the desktop screen or the screen of the application selected by the user from the OS (Operating System), etc. If the application selected by the user is the remote conferencing application 42, the screen generated by the remote conferencing application 42 (images of each location, images of documents, etc.) is obtained.

[0090] The audio acquisition unit 15 acquires audio output from the microphone or earphones of the terminal device 10 (including audio data received from the remote conferencing application 42 during a remote conference). The audio acquisition unit 15 can acquire audio even if the output audio is muted. The user does not need to perform any operations such as selecting the remote conferencing application 42 regarding audio data; the audio acquisition unit 15 can acquire the audio that the terminal device 10 can output via the OS or API (Application Interface). As a result, audio data received by the remote conferencing application 42 from other locations 101 is also acquired. If the remote conferencing application 42 is not running or a remote conference is not in progress, the information recording application 41 may not be able to acquire audio data. Note that the audio acquired by the audio acquisition unit 15 does not include the audio collected by the terminal device 10, but only the output audio data. This is because the meeting device 60 collects audio separately.

[0091] The device communication unit 16 communicates with the meeting device 60 using a USB cable or the like. The device communication unit 16 may communicate with the meeting device 60 using wireless LAN, Bluetooth (registered trademark), etc. The device communication unit 16 receives panoramic images and speaker images from the meeting device 60 and transmits the audio data acquired by the audio acquisition unit 15 to the meeting device 60. The device communication unit 16 receives the audio data synthesized by the meeting device 60.

[0092] The video storage unit 17 combines the panoramic image and speaker image received by the device communication unit 16, and the application screen acquired by the application screen acquisition unit 14, to create a combined image. The video storage unit 17 also repeatedly connects the created combined images in chronological order to create a combined image video, and combines the synthesized audio data with the combined image video to create a combined image video with audio.

[0093] The audio data processing unit 18 either extracts the audio data combined into the combined image video, or requests the information processing system 50 to convert the synthesized audio data received from the meeting device 60 into text data.

[0094] The recording and playback unit 19 plays back the combined image video. The combined image video is stored in the terminal device 10 during recording and then uploaded to the information processing system 50.

[0095] When the remote conference ends, the upload unit 20 transmits the combined image and video to the information processing system 50.

[0096] The editing processing unit 21 performs editing of the combined image video (such as deleting parts or splicing them together) in response to user operations.

[0097] The search unit 22 accepts a search request for text data using keywords, searches the text data, and displays the search results.

[0098] Figure 9 shows the video recording information stored in the information storage unit 1001. The video recording information includes items such as meeting ID, recording ID, update date and time, title, upload, and storage location. When a user logs into the information processing system 50, the information recording application 41 downloads meeting information from the meeting information storage unit 5001 of the information processing system 50. The meeting ID and other information included in the meeting information are reflected in the video recording information. The video recording information in Figure 9 is held by a terminal device 10 operated by a particular user.

[0099] The meeting ID is identification information that identifies a remote meeting that has been held. The meeting ID is assigned when the remote meeting is scheduled in the meeting management system 9, or it is assigned by the information processing system 50 upon request from the information recording application 41.

[0100] The recording ID is identification information that identifies the combined video image recorded during a remote meeting. The recording ID is assigned by the meeting device 60, but it may also be assigned by the information recording application 41 or the information processing system 50. The same meeting ID may be assigned to a different recording ID if the recording of the remote meeting was stopped midway through but resumed for some reason.

[0101] • The update date and time is the date and time when the combined image / video was updated (recording ended). If the combined image / video was edited, it will be the date and time of editing.

[0102] The title is the name of the meeting. This may be set when registering the meeting in the meeting management system 9, or it may be set arbitrarily by the user.

[0103] • Upload indicates whether or not the combined image video has been uploaded to the information processing system 50.

[0104] The storage location indicates the location (URL or file path) where the combined image / video and text data are stored in the storage service system 70. Therefore, users can view the uploaded combined image / video at their discretion. Note that the combined image / video and text data are saved with separate filenames, for example, following the URL.

[0105] <<Meeting Devices>> Let's return to Figure 8 for explanation. The meeting device 60 includes a communication unit 61, a panoramic image creation unit 62, a speaker image creation unit 63, a location determination processing unit 64, a speech synthesis unit 65, a text conversion request unit 66, a device connection unit 67, and a sound collection unit 68. Each of these parts of the meeting device 60 is a function or means of functioning that is implemented in hardware by any of the components shown in Figure 5. However, each of these parts may also be implemented by operating according to instructions from the CPU 611 that follow a program expanded from the ROM 612 into the DRAM 614.

[0106] The communication unit 61 transmits and receives various types of information with the information processing system 50. The communication unit 61 can also communicate with the voice recognition service system and the storage service system 70.

[0107] The device connection unit 67 accepts audio input from the terminal device 10. The device connection unit 67 communicates with the terminal device 10 using a USB cable or the like. The device connection unit 67 may communicate with the terminal device 10 using wireless LAN, Bluetooth (registered trademark), etc. The meeting device 60 may be equipped with a speaker capable of outputting audio B from another location acquired by the device connection unit 67. The device connection unit 67 corresponds to the audio input unit 8 from another location in Figure 2.

[0108] The panoramic image creation unit 62 creates a panoramic image. The speaker image creation unit 63 creates a speaker image. The methods for creating these images are explained in Figures 6 and 7.

[0109] The location determination processing unit 64 divides the local voice A and the other location voice B according to a predetermined rule, and determines the location from which the local voice A or the other location voice B was spoken based on the local sound pressure information A' of the divided local voice A and the other location sound pressure information B' of the other location voice B. Further details will be described later.

[0110] The speech synthesis unit 65 synthesizes the speech transmitted from the terminal device 10 with the speech collected by the sound collection unit 68. This combines speech spoken at other locations 101 with speech spoken at the local location 102.

[0111] The sound collection unit 68 converts the audio signal A of the local location, acquired by the microphone of the meeting device 60, into audio data (digital). This allows the content spoken by users and participants at the terminal device 10 to be collected. The local location audio A collected by the sound collection unit 68 is preferably transmitted to the terminal device 10 via the device connection unit 67. This allows the user to use the meeting device 60 in the same way as a typical external microphone device. The sound collection unit 68 (and may also include the audio acquisition unit 15) corresponds to the local location audio input unit 7 in Figure 2.

[0112] Furthermore, the meeting device 60 has a sound collection unit 68 and a device connection unit 67, respectively, which enables the separate acquisition of local audio A and audio B from other locations. Therefore, it becomes possible to separately acquire local sound pressure information A' and other location sound pressure information B'.

[0113] The text conversion request unit 66 transmits synthesized speech C to the information processing system 50 via the communication unit 61 and requests real-time speech recognition (conversion to text data).

[0114] <<Information Processing System>> The information processing system 50 includes a communication unit 51, an authentication unit 52, a screen generation unit 53, a conference information acquisition unit 54, a speech recognition unit 55, a speech recording unit 56, a speech recognition result recording unit 57, a location determination result recording unit 58, and a conference record confirmation unit 59. Each of these units in the information processing system 50 is a function or means of functioning, realized by any of the components shown in Figure 4 operating according to instructions from the CPU 501 following a program deployed from the HD 504 to the RAM 503. The information processing system 50 also has a storage unit 5000 constructed from the HD 504 and other components shown in Figure 4. The storage unit 5000 contains a conference information storage unit 5001 and a recording information storage unit 5002.

[0115] The communication unit 51 sends and receives various types of information with the terminal device 10. For example, the communication unit 51 may send a list of remote meetings to the terminal device 10 or receive a voice data recognition request from the terminal device 10.

[0116] The authentication unit 52 authenticates the user operating the terminal device 10. The authentication unit 52 authenticates the user, for example, by checking whether the authentication information (user ID and password) included in the authentication request received by the communication unit 51 matches the authentication information it has previously stored. The authentication information may be an IC card number, biometric authentication information such as a face or fingerprint, etc. The authentication unit 52 may also authenticate using an external authentication system or an authentication method such as OAuth.

[0117] The screen generation unit 53 generates screen information to be displayed by the terminal device 10. When the terminal device 10 runs a native application, the screen information is held by the terminal device 10, and the information to be displayed is transmitted in XML or the like. When the terminal device 10 runs a web application, the screen information is created using HTML, XML, CSS (Cascade Style Sheet), and JavaScript (registered trademark), etc.

[0118] The meeting information acquisition unit 54 acquires information about remote meetings from the meeting management system 9 using each user's account or a system account assigned to the information processing system 50. The meeting information acquisition unit 54 can acquire a list of remote meetings that users belonging to the tenant have viewing rights to. Since remote meetings are assigned a meeting ID, the remote meeting and its record information are associated using the meeting ID.

[0119] The speech recognition unit 55 converts the synthesized speech C, which has been requested to be converted to text data from the terminal device 10 or meeting device 60, into text data using an external speech recognition service. The speech recognition unit 55 may also perform the conversion itself.

[0120] The voice recording unit 56 records at least synthesized speech C. The speech recognition result recording unit 57 stores the speech recognition result D. The location determination result recording unit 58 stores location identification information E. The location determination result recording unit 58, the speech recognition result recording unit 57, and the voice recording unit 56 may be stored in the information processing system 50 or in the storage service system 70. That is, the location determination result recording unit 58, the speech recognition result recording unit 57, and the voice recording unit 56 may function as passive storage, or they may function in cooperation with the recording information storage unit 7001 of the storage service system 70. In this embodiment, the latter will be mainly described.

[0121] The synthesized speech C, speech recognition result D, and location identification information E are stored in a segmented state, corresponding to each other. The recognition result string is a string generated at each segment of speech recognition. Where speech recognition is segmented is set as appropriate, for example, at a silent period or a certain time interval.

[0122] The meeting record verification unit 59 synchronously acquires the location identification information E, the speech recognition result D, and the synthesized speech C, and presents them to the user to accept confirmation of the location's determination.

[0123] With this configuration, the meeting device 60 can utilize remote conferencing using widely available remote communication systems, and can capture audio in a natural manner and identify the source of the speaker without using multiple dedicated microphones for audio collection and speaker identification.

[0124] <<Conference Information Storage Unit>> Figure 10 shows an example of meeting information stored in the meeting information storage unit 5001, which is managed by the meeting information acquisition unit 54. The meeting information acquisition unit 54 can use the above account to obtain a list of remote meetings that the user belonging to the tenant has permission to view. In this embodiment, remote meetings are used as an example, but the list of remote meetings also includes meetings held in only one conference room.

[0125] Meeting information is managed using a meeting ID, which is associated with participants, title (meeting name), start time, end time, location, etc. These are just examples of meeting information; meeting information may include other details as well.

[0126] The meeting ID is the identification information for the meeting.

[0127] • The tenant ID is the identifying information for the tenant where the meeting is being held.

[0128] The title is the subject or agenda of the meeting.

[0129] The organizer is the person who organized the meeting.

[0130] • Participants is a list of those invited to the meeting.

[0131] • Users with viewing privileges are a list of users who have access to meeting resources, including combined images and videos.

[0132] • Ad-hoc participants are a list of guest participants.

[0133] • The location is information about the meeting room, such as its name.

[0134] • The start time is the scheduled time when the meeting will begin.

[0135] • The end time is the scheduled time when the meeting will conclude.

[0136] The meeting creator is the user ID, etc., who registered the meeting information.

[0137] The password is the password that participants use to log in to the meeting.

[0138] The location is where the meeting is held, such as a conference room, branch office name, or building.

[0139] As shown in Figures 9 and 10, the combined video footage recorded during the meeting is identified by the meeting ID.

[0140] Figure 11 shows the recording information stored in the recording information storage unit 5002. The recording information contains a list of combined video recordings made by all users belonging to the tenant. The recording information includes items such as meeting ID, recording ID, update date and time, title, and storage location. These items can be the same as in Figure 9.

[0141] <<Storage Service System>> The storage service system 70 can be any service system that stores recorded information. The recorded information storage unit 7001 stores recorded information (combined image video, text data). Figure 12 illustrates the structure of the text data stored in the recorded information storage unit 7001.

[0142] Figure 12 illustrates the structure of text data stored in the storage service system 70. As shown in Figure 12, the text data is associated with the following items: ID, time, recognition result string, voice data, and location identification information E. The text data is stored associated with the meeting ID.

[0143] The ID is identification information assigned when local audio A and other location audio B are split according to a predetermined rule. The predetermined rule is set in the meeting device 60 (and at least one of the speech recognition service system 80), and may include, for example, splitting after a certain period of silence, forcibly splitting after a certain period of time even without silence, or splitting sentence by sentence as detected by morphological analysis.

[0144] • `time` is the speaking time based on the duration since the recording started. Since the so-called time is also saved when the recording starts, the time when the text was spoken (absolute time) can also be determined.

[0145] The recognition result string is a portion of the text data converted by speech recognition from the divided synthesized speech C. Synthesized speech C is the audio data from which the recognition result string was converted.

[0146] The audio data is synthesized voice C (already split) created by combining voice A from the local site and voice B from another site after the site has been identified.

[0147] • Location identification information E is the identification information of the location from which the voice data was spoken, determined based on the sound pressure of the local location sound pressure information A' and the sound pressure information B' of another location. For example, location identification information E may represent 1 as the local location and 2 as another location.

[0148] Since time is associated with the recognition result string in this way, when the text data matches the search, the information recording app 41 can play the combined image video from the playback time corresponding to this time.

[0149] <Screen transition> Next, with reference to Figures 13 to 16, several screens displayed by the terminal device 10 during a remote conference will be described. Figure 13 is the initial screen 200 displayed after login by the information recording application 41 running on the terminal device 10. The user of the terminal device 10 connects the information recording application 41 to the information processing system 50. When the user enters authentication information and successfully logs in, the initial screen 200 of Figure 13 is displayed.

[0150] The initial screen 200 has a fixed display button 201, a front view change button 202, a panoramic image 203, one or more speaker images 204a to 204c (hereinafter referred to as speaker image 204 when not distinguished), and a recording start button 205. If the meeting device 60 is already running and capturing images of the surroundings when the user logs in, the panoramic image 203 and speaker images 204 created by the meeting device 60 are displayed on the initial screen 200. Therefore, the user can decide whether or not to start recording while viewing these images. If the meeting device 60 is not running (not capturing images), the panoramic image 203 and speaker images 204 are not displayed.

[0151] The information recording application 41 may display speaker images 204 of all participants based on all faces detected from the panoramic image 203, or it may display only the speaker images 204 of the N people who have spoken most recently. Figure 13 shows an example where up to three speaker images 204 are displayed. There may be no speaker images 204 until a participant speaks (one is added each time a participant speaks), or three speaker images 204 of participants facing a predetermined direction may be displayed (they are swapped depending on the participant's speech).

[0152] If no one is speaking, such as immediately after the meeting device 60 is activated, a speaker image 204 will be created in a predetermined direction within the horizontal 360° (e.g., 0°, 120°, 240°). If a fixed display setting (described later) is enabled, that setting will take precedence.

[0153] The fixed display button 201 is a button that allows the user to perform an operation to fix a certain area of ​​the panoramic image 203 as the speaker image 204 and zoom in on it.

[0154] The front view change button 202 is a button for the user to change the front view of the panoramic image 203 (since the panoramic image captures 360° horizontally, the right and left edges coincide). The user can slide the panoramic image 203 left or right with a pointing device to determine which participant is in the front view. The user's operation is sent to the meeting device 60, which changes the angle that is considered the front view out of the 360° horizontal direction to create a panoramic image and sends it to the terminal device 10.

[0155] When the user presses the recording start button 205, the information recording application 41 displays the recording settings screen 210 shown in Figure 14.

[0156] Figure 14 shows an example of the recording settings screen 210 displayed by the information recording application 41. On the recording settings screen 210, the user can set whether to record (include in recording) the panoramic image and speaker image created by the meeting device 60, as well as the desktop screen or running application screen of the terminal device 10. If the information recording application 41 does not record either the panoramic image and speaker image, or the desktop screen or running application screen, only the audio (audio output by the terminal device 10 + audio collected by the meeting device 60) will be recorded.

[0157] The camera toggle button 211 is a button that switches the recording of panoramic images and speaker images created by the meeting device 60 on and off. The camera toggle button 211 may also be configured to record panoramic images and speaker images individually.

[0158] The PC screen toggle button 212 is a button that switches the recording of the desktop screen of the terminal device 10 and the screen of an application running on the terminal device 10 on and off. When the PC screen toggle button 212 is on, the desktop screen is recorded.

[0159] If a user wants to record an app screen, they further select an app from the app selection field 213. The app selection field 213 displays the names of apps currently running on the terminal device 10 in a dropdown format. The user can select the app to record. This app name is obtained by the information recording app 41 from the OS. The information recording app 41 can display only apps that have a UI (screen) among the running apps. The remote conferencing app 42 may be included in the selected apps. Therefore, the information recording app 41 can also record the materials displayed by the remote conferencing app 42 and the participants at each location in video. In addition, the apps displayed in the dropdown are diverse, including presentation apps, word processing apps, spreadsheet apps, and web browser apps. Therefore, the user can flexibly select the app screens to include in the combined video.

[0160] Furthermore, when recording on an app-by-app basis, users can select multiple apps. The information recording app 41 can record the screens of all selected apps.

[0161] If both the camera toggle button 211 and the PC screen toggle button 212 are turned off, the recording content confirmation window 214 will display "Only audio will be recorded." This audio consists of the audio output by the terminal device 10 (audio received by the remote conferencing application 42 from other locations 101) and the audio collected by the meeting device 60. In other words, if a remote conference is being held, the audio from the remote conferencing application 42 and the audio from the meeting device 60 will be saved regardless of image recording. However, the user may selectively stop saving the audio from the remote conferencing application 42 and the audio from the meeting device 60 in their settings.

[0162] Depending on the on / off combination of the camera toggle button 211 and the PC screen toggle button 212, the combined image video is recorded as follows. The combined image video is also displayed in real time in the recording content confirmation window 214.

[0163] If the camera toggle button 211 is on and the PC screen toggle button 212 is off, the panoramic image captured by the meeting device 60 and the speaker image will be displayed in the recording content confirmation window 214.

[0164] If the camera toggle button 211 is off and the PC screen toggle button 212 is on (screen is also selected), the desktop screen or the screen of the selected application will be displayed in the recording content confirmation window 214.

[0165] When the camera toggle button 211 is on and the PC screen toggle button 212 is on, the recording content confirmation window 214 displays the panoramic image and speaker image captured by the meeting device 60, as well as the desktop screen and the screen of the selected application, side by side.

[0166] Therefore, there are cases where the panoramic image, speaker image, and app screen are not recorded, or where none of the panoramic image, speaker image, and app screen are recorded at all. However, in this embodiment, for convenience, the images created by the information recording app 41 are referred to as a combined image video.

[0167] Furthermore, the recording settings screen 210 has a checkbox 215 along with the message, "Automatically transcribe the recording after uploading it." The recording settings screen 210 also has a "Start Recording Now" button 216. When the user checks the checkbox 215, the recorded video is attached with text data containing the spoken words from the remote meeting. In this case, after the recording is finished, the information recording application 41 uploads the audio to the information processing system 50 along with a request for conversion to text data. Also, when the user presses the "Start Recording Now" button 216, the recording screen is displayed.

[0168] Figure 15 shows an example of the meeting list screen 230 displayed by the information recording application 41. The meeting list screen 230 is a list of meetings, and can display a list of recorded information from remote meetings. It also includes meetings held only within a specific meeting room, regardless of whether they were remote meetings. The meeting list screen 230 displays meeting information that the logged-in user has permission to view in the meeting information storage unit 5001. Information from video recordings stored in the information storage unit 1001 may be further integrated.

[0169] The meeting list screen 230 is displayed when the user selects the meeting list tab 231 on the initial screen 200 in Figure 13. The meeting list screen 230 displays a list 236 of recorded information that the user has permission to view. The meeting creator (minutes creator) can set viewing permissions for participants. The meeting list may be a list of saved recorded information, or a list of meeting schedules or meeting data.

[0170] The meeting list screen 230 has the following items: checkbox 232, update date and time 233, title 234, and status 235.

[0171] Checkbox 232 accepts the selection of recording files. Checkbox 232 is used when the user wants to delete multiple recording files at once.

[0172] • The update time 233 indicates the start and end times of the combined image video recording. If it has been edited, the edit time can be used instead.

[0173] • The title is 234, which is the title of the meeting (agenda, etc.). It can be copied from the meeting information or set by the user.

[0174] Status 235 indicates whether the combined image / video has been uploaded to the information processing system 50. If it has not been uploaded, "Local PC" is displayed; if it has been uploaded, "Uploaded" is displayed. If it has not been uploaded, the upload button is displayed. If there are unuploaded combined image / videos, the information recording application 41 should automatically upload them when the user logs into the information processing system 50.

[0175] When the user selects a title or other item from the list of combined images and videos 236 using the pointing device, the information recording application 41 displays a recording playback screen, but this is omitted in this embodiment. On the recording playback screen, it is possible to play back the combined images and videos.

[0176] Ideally, users should be able to filter meetings by update date, title, keywords, etc. Furthermore, if the number of displayed meetings is large and it is difficult to find the relevant one, a search function should be available allowing users to input terms to narrow down the recorded information based on keywords found in meeting statements or titles. This search function would enable users to quickly find the desired recorded information, even when dealing with a large volume of records. Additionally, the meeting list screen should allow users to sort meetings by update date or title.

[0177] <Example of a screen displayed by the meeting minutes verification department> Figure 16 shows an example of a meeting record confirmation screen 240, which is presented by the meeting record confirmation unit 59 and displayed by the terminal device 10.

[0178] The meeting record confirmation screen 240 has any number of segment display units 130-1 to 130-n. Hereinafter, any segment display unit among the segment display units 130-1 to 130-n will simply be referred to as segment display unit 130.

[0179] Each segment display unit 130 includes at least a speech recognition result display unit 133, a location identification information display unit 131, and a synthesized speech playback unit 132. The speech recognition result display unit 133 displays the speech recognition result D. The location identification information display unit 131 displays location identification information E corresponding to the speech recognition result D. The synthesized speech playback unit 132 has at least a play button for playing the audio, and the user can play the synthesized speech C corresponding to the speech recognition result D displayed on the speech recognition result display unit 133 by pressing the play button. The synthesized speech playback unit 132 may also include functions for pause, double-speed playback, changing the playback position using a seek bar, and skipping to an arbitrary time.

[0180] By displaying the speech recognition result D in association with the corresponding location identification information E, users can better understand the meeting minutes and prevent misrecognition due to differences in the speaking locations. The ability to play back synthesized speech C allows users to review the meeting minutes more easily.

[0181] In Figure 16, the presence of a location identification information display unit 131 and synthesized voice playback unit 132 on the right and on the left indicates the difference between locations. For example, if the statement is made at the user's own location, the location identification information display unit 131 and synthesized voice playback unit 132 are displayed on the right, and if the statement is made at another location, they are displayed on the left. This allows the user to determine the location at a glance.

[0182] It is preferable that the information displayed in the speech recognition result display unit 133 and the location identification information display unit 131 can be edited by the user at any time. Furthermore, it is preferable that any part of the segment display units 130-1 to n can be deleted by the user. This allows the user to arbitrarily correct or delete the recognized string if there is an error in the speech recognition result or if there is unnecessary text data in the meeting minutes.

[0183] When a user edits (changes) the location identification information display unit 131, the display control unit 13 changes the arrangement of the location identification information display unit 131 and the synthesized voice playback unit 132 from left to right or right to left, according to the edited location. This screen display change processing may also be performed by the screen generation unit 53. When a user edits the location identification information E, the meeting record confirmation unit 59 receives the edited content from the terminal device 10 and changes the location identification information E in the text data shown in Figure 12.

[0184] <Decision on the location> Figure 17 is a diagram illustrating the location determination criteria. The upper and lower figures of Figure 17(a),(b), and(c) show examples of sound pressure information for local location audio A and other location audio B, respectively. Sound pressure is the amplitude of the waveform in Figure 17(a),(b), and(c), and the input voltage V IN It is obtained by converting it to decibel full scale (dBFS) using equation (1). In other words, the decibel full scale is the sound pressure. Figures 17(a), (b), and (c) are obtained in this way.

[0185]

number

[0186] The peak value method uses the peak value (maximum value) of an audio waveform as a representative value of the sound pressure information of a given audio data. The effective value (RMS value) is used when the input voltage of a given audio is V IN The effective value V of the waveform IN This method uses the value obtained by converting -rms to sound pressure (dBFS) using the above formula (1) as a representative value of the sound pressure information. The effective value can be calculated using formula (2).

[0187]

number

[0188] Next, we will show examples of conditions for the location determination processing unit 64 to determine a location. As for the conditions for determining a location, condition 2 is preferable to condition 1, which will be explained below, but it is not limited to these, and the optimal conditions can be selected for each system. In the location determination conditions below, the sound pressure information of the local location A' is simply written as A', and the sound pressure information of other locations B' is simply written as B', but the peak value, effective value (RMS value), etc. mentioned above can be used for these, respectively.

[0189] <<Conditions for determining a base location 1-1>> (1) The base determination processing unit 64 compares the values ​​of A' and B' and determines that a message was made at its own base if A' is greater than B'.

[0190] (2) The location determination processing unit 64 compares the values ​​of A' and B', and if B' is greater than A', it determines that a message was made at another location.

[0191] (3) The location determination processing unit 64 determines "NA (unknown)" if the sound pressure is the same.

[0192] The above judgment examples are represented in a flowchart as shown in Figure 18. Generally, in the case of NA, the location identification information E is unknown, so the recognition result string is also deleted. However, the location determination processing unit 64 may retain the recognition result string, or it may switch whether to retain the recognition result string that was determined to be NA depending on the location determination conditions.

[0193] Figure 18 is an example of a flowchart illustrating the method for determining a base based on base determination condition 1-1.

[0194] As described above, the base determination processing unit 64 divides the voice into the self-base voice A and the other-base voice B according to a predetermined rule, and determines the magnitude relationship between the values of A' and B' (S101).

[0195] When A' > B', the base determination processing unit 64 determines that there is a speech at the self-base (S102).

[0196] When A' < B', the base determination processing unit 64 determines that there is a speech at the other base (S103).

[0197] When A' = B', the base determination processing unit 64 determines that it is unknown (S104).

[0198] <<Base determination condition 1-2>> (1) The base determination processing unit 64 compares the values of A' and B', and if A' is greater than B', it determines that there is a speech at the self-base.

[0199] (2) The base determination processing unit 64 compares the values of A' and B', and if B' is greater than or equal to A', it determines that there is a speech at the other base (when the sound pressure is the same, it is determined as "other base"). According to the base determination condition 1-2, by comparing A' and B' and determining the voice with the larger value as the base, it is possible to make the voice recognition result and the base identification information E coincide with a high probability. In addition, since the voice of the other base is the voice received via the remote communication system, the voice may be slightly reduced by the noise cancellation function of the Web conferencing system. Therefore, even if the actual volume of speaking is the same, the acquired sound pressure may be reduced. Therefore, when A' and B' have the same sound pressure according to the base determination condition 1-2, misjudgment of the base can be reduced by determining it as the other base.

[0200] FIG. 19 is an example of a flowchart for explaining the base determination method according to the base determination condition 1-2.

[0201] As described above, the base determination processing unit 64 divides the voice into the self-base voice A and the other-base voice B according to a predetermined rule, and determines whether A' is greater than B' (S111).

[0202] The base determination processing unit 64 determines that a message was made at its own base if A' > B' (S112).

[0203] The location determination processing unit 64 determines that a message was made at another location if A'≦B' (S113).

[0204] <<Conditions for determining a base location 2>> (1) The location determination processing unit 64 determines that a message was made at another location if the value of B' is greater than or equal to the noise threshold X, and B' is greater than the value of A'.

[0205] (2) The location determination processing unit 64 determines that a message was made at its own location if the value of A' is greater than or equal to the noise threshold X, and A' is greater than the value of B'.

[0206] (3) If the value of either A' or B' is less than the noise threshold X, the location determination processing unit 64 determines that there were no statements at either location (N / A (unknown)).

[0207] According to condition 2, the location determination processing unit 64 first compares the magnitudes of A' and B' with the noise threshold X, and then compares the magnitudes of A' and B' again to return the location with the larger magnitude as location identification information E. This prevents misidentification of locations due to the influence of environmental noise in environments with environmental noise.

[0208] The noise threshold X can be selected to an appropriate value depending on the system, but it is preferable to use a value of around -40 dBFS when using the peak value on a full decibel scale as sound pressure information, and around -50 dBFS when using the RMS value on a full decibel scale as sound pressure information.

[0209] Figure 20 is an example of a flowchart illustrating the method for determining a base based on base determination criterion 2.

[0210] The location determination processing unit 64 divides the audio into local location audio A and other location audio B according to a predetermined rule. The location determination processing unit 64 determines whether the value of B' is greater than or equal to the noise threshold X (S121). If the determination in step S121 is Yes, the location determination processing unit 64 determines whether the value of B' is greater than A' (S122).

[0211] If the decision in step S122 is Yes, the location determination processing unit 64 determines that a statement was made at another location (S123).

[0212] If the determination in step S121 or S122 is No, the site determination processing unit 64 determines whether the value of A' is greater than or equal to the noise threshold X (S124).

[0213] If the decision in step S124 is Yes, the site determination processing unit 64 determines whether the value of A' is greater than B' (S125).

[0214] If the decision in step S125 is Yes, the base determination processing unit 64 determines that a statement was made at its own base (S126).

[0215] If the determination in step S124 or S125 is No, the location determination processing unit 64 determines that the location is unknown (S127).

[0216] Note that the order in which decisions are made regarding other locations and your own location can be reversed.

[0217] <<Example of judgment>> This section explains an example of how to determine a site by applying site determination conditions 1 and 2 to the sound pressure level shown in Figure 16.

[0218] In Figure 17(a), when using location determination condition 1-1 and peak values ​​as sound pressure information, the location is determined to be the local location as location identification information E.

[0219] In Figure 17(a), when using site determination conditions 1-2 and using the RSM value as sound pressure information, the site is determined to be the local site as site identification information E.

[0220] In Figure 17(a), if the noise threshold is set to -40 dBFS, and site determination condition 2 is used, and the peak value is used as sound pressure information, the site identification information E will be determined to be the local site.

[0221] In Figure 17(a), when the noise threshold is set to -50 dBFS, site determination condition 2 is used, and the effective value (RSM value) is used as sound pressure information, the site is determined to be the local site as site identification information E.

[0222] By utilizing base determination conditions 1 and 2, it became possible to determine a base where a statement is presumed to have been made as base identification information E.

[0223] In Figure 17(b), when using location determination condition 1-1 and peak values ​​as sound pressure information, the location is determined to be another location as location identification information E.

[0224] In Figure 17(b), when using site determination condition 1-2 and using the RSM value as sound pressure information, the site is determined to be the local site.

[0225] In Figure 17(b), if the noise threshold is set to -40 dBFS, and location determination condition 2 is used, and the peak value is used as sound pressure information, then location identification information E will be determined to be another location.

[0226] In Figure 17(b), when the noise threshold is set to -50 dBFS, site determination condition 2 is used, and the effective value (RSM value) is used as sound pressure information, the site identification information E is determined to be the local site.

[0227] By using location determination condition 2, the location determination results became less susceptible to noise.

[0228] In Figure 17(c), when using location determination condition 1-1 and peak values ​​as sound pressure information, the location identification information E is determined to be the local location.

[0229] In Figure 17(c), when using site determination conditions 1-2 and using the RSM value as sound pressure information, the site identification information E is determined to be the local site.

[0230] In Figure 17(c), if the noise threshold is set to -40 dBFS, and site determination condition 2 is used, and the peak value is used as sound pressure information, the site identification information E will be determined to be the local site.

[0231] In Figure 17(c), when the noise threshold is set to -50 dBFS, site determination condition 2 is used, and the effective value (RSM value) is used as the sound pressure information, the site identification information E is determined to be N / A (no site identification information).

[0232] By using RMS values ​​as sound pressure information, the site determination results have become less susceptible to the influence of sudden noises.

[0233] <Mapping of location identification information E, speech recognition result D, and synthesized speech C> In Figure 12, an ID is used to associate the location identification information E, the speech recognition result D, and the synthesized speech C. However, methods such as a queue system and a timestamp system can be considered for associating the same ID with the location identification information E, the speech recognition result D, and the synthesized speech C.

[0234] <<Queue System>> Figure 21 illustrates how the information processing system 50 associates location identification information E, speech recognition result D, and synthesized speech C using a queue method. A queue is a data structure in which elements are arranged in a line in the order they are received, and elements are retrieved in the order they were received. In the queue method, the information processing system 50 assigns the same ID to location identification information E, speech recognition result D, and synthesized speech C according to the order in which they are input to the location determination result recording unit 58, speech recognition result recording unit 57, and speech recording unit 56, respectively.

[0235] The meeting record confirmation screen 240 displays the location identification information E-1, speech recognition result D-1, and synthesized speech C-1 in association with each other in the first segment display unit 130, which is segment display unit 130-1. The meeting record confirmation screen 240 displays the location identification information En, speech recognition result Dn, and synthesized speech Cn in association with each other in the nth segment display unit 130, which is segment display unit 130-n.

[0236] The queue method uses a simple algorithm to display to the user the location identification information E, speech recognition results D, and synthesized speech C in association with each other.

[0237] <<Timestamp method>> Figure 22 illustrates how the information processing system 50 associates location identification information E, speech recognition result D, and synthesized speech C using a timestamp method. The timestamp method is a method of adding the same timestamp as metadata to location identification information E, speech recognition result D, and synthesized speech C. A timestamp is a method that allows for reliable and easy verification of data identity based on timestamp information F, which is recorded in the timestamp, such as time or hash value. The meeting device 60 adds the timestamps to location identification information E and synthesized speech C. The information processing system 50 adds the timestamp to speech recognition result D.

[0238] The information processing system 50 assigns the same ID to the location identification information E, speech recognition result D, and synthesized voice C, which all have the same timestamp information F. The meeting record confirmation screen 240 displays the location identification information E, speech recognition result D, and synthesized voice C, which all have the same timestamp information F, in association with each other.

[0239] The meeting record confirmation screen 240 displays the data with timestamp information F-1, namely the location identification information E-1, speech recognition result D-1, and synthesized speech C-1, in association with each other, in the segment display unit 130-1, which is the first segment display unit 130. The meeting record confirmation screen 240 displays the data with timestamp information Fn, namely the location identification information En, speech recognition result Dn, and synthesized speech Cn, in association with each other, in the segment display unit 130-n, which is the nth segment display unit 130.

[0240] The timestamp method allows for the disclosure of site identification information E, speech recognition results D, and synthesized speech C to the user using a simple algorithm. Furthermore, even if data delays cause the order in which site identification information E, speech recognition results D, and synthesized speech C enter the site determination result recording unit 58, speech recognition result recording unit 57, and speech recording unit 56 to be reversed, the site identification information E, speech recognition results D, and synthesized speech C can be reliably synchronized.

[0241] When generating a timestamp, it is preferable to generate it at the time of acquiring local audio A or other-site audio B. However, the meeting device 60 may also generate it at the time of generating synthesized speech C, at the time of acquiring local sound pressure information A' or other-site sound pressure information B', or at the time of generating site identification information E. The timing of timestamp generation is not limited to these, as long as it is at a time when the site identification information E, speech recognition result D, and synthesized speech C can be associated.

[0242] <Operating Procedure> Next, we will explain the operations and processes performed by the record information creation system 100 based on the above configuration.

[0243] <<Saving combined images and videos>> Referring to Figure 23, the process of saving the combined image video will be explained. Figure 23 is an example of a sequence diagram showing the procedure by which the information recording application 41 records a panoramic image, a speaker image, and the application screen. Figure 23 also explains the process of real-time speech recognition.

[0244] S1: The user launches the information recording application 41 on the terminal device 10 and connects the terminal device 10 (information recording application 41) to the information processing system 50. If the access token has expired, the display control unit 13 displays the login screen. The user enters authentication information (e.g., user ID, password) to log in to the tenant into the information recording application 41. The operation reception unit 12 of the information recording application 41 accepts the input.

[0245] S2: The communication unit 11 of the information recording application 41 sends a login request to the information processing system 50, specifying the authentication information.

[0246] S3: The communication unit 51 of the information processing system 50 receives a login request, and the authentication unit 52 authenticates the user based on the authentication information. Here, we assume that authentication was successful. The communication unit 51 of the information processing system 50 sends the access token 1 to the information recording application 41. Although not explicitly shown in the diagram, the communication unit 51 will subsequently attach the access token 1 to communications with the information processing system 50. The access token 1 is associated with the permissions of the logged-in user.

[0247] S4: Similarly, the user also logs into the storage service system 70. This is because the recorded information is stored in the storage service system 70. The user enters authentication information (e.g., user ID, password) to log into the storage service system 70. The operation reception unit 12 of the information recording application 41 receives the input.

[0248] S5: The communication unit 11 of the information recording application 41 sends a login request to the information processing system 50, specifying the authentication information.

[0249] S6: The communication unit 51 of the information processing system 50 receives a login request, and since it is a login request to the storage service system 70, it forwards the login request to the storage service system 70.

[0250] S7: The storage service system 70 authenticates the user based on the authentication information. Here, we assume that authentication was successful. The storage service system 70 sends the access token 2 to the information processing system 50.

[0251] S8: The communication unit 51 of the information processing system 50 receives the access token 2 and sends it to the information recording application 41. Although not explicitly shown in the diagram, the communication unit 51 subsequently attaches the access token 2 to its communication with the storage service system 70. The access token 2 is associated with the permissions of the logged-in user.

[0252] S21: Next, the user operates the remote conferencing application 42 to start a remote conference. Here, it is assumed that the remote conferencing applications 42 at the local site 102 and the other site 101 have started a remote conference. The remote conferencing application 42 at local site 102 transmits the image captured by the camera and the audio collected by the microphone of the terminal device 10 to the remote conferencing application 42 at the other site 101. The remote conferencing application 42 at the other site 101 displays the received image on its display and outputs the received audio through its speaker. Similarly, the remote conferencing application 42 at the other site 101 transmits the image captured by the camera and the audio collected by the microphone of the terminal device 10 to the remote conferencing application 42 at local site 102. The remote conferencing application 42 at local site 102 displays the received image on its display and outputs the received audio through its speaker. Each remote conferencing application 42 repeats this process to realize a remote conference.

[0253] S22: The user configures the recording settings on the recording settings screen 210 of the information recording application 41 shown in Figure 16. The operation reception unit 12 of the information recording application 41 accepts the settings. Here, it is assumed that both the camera toggle button 211 and the PC screen toggle button 212 are turned on.

[0254] If the user has already scheduled a remote meeting, a list of remote meetings, as shown in Figure 15, will be displayed, and the user can select the remote meeting to which the combined image video will be associated. Since the user is logged into the information processing system 50, the information processing system 50 identifies the remote meetings that the logged-in user has access to. The information processing system 50 sends the list of identified remote meetings to the terminal device 10, and the user selects a remote meeting that is currently in session or will be held soon. This determines the information related to the remote meeting, such as the meeting ID.

[0255] Furthermore, users can create a meeting when creating a combined image video, even if they haven't scheduled a remote meeting in advance. The following describes how the information recording application 41 creates a meeting when creating a combined image video and obtains the meeting ID from the information processing system 50.

[0256] S23: The user instructs the information recording application 41 to start recording (by pressing the "Start Recording Now" button 216). The operation reception unit 12 of the information recording application 41 receives the instruction. The display control unit 13 displays the recording screen.

[0257] S24: Since remote conferencing is not selected (because the conferencing ID has not been determined), the communication unit 11 of the information recording application 41 sends a remote conferencing creation request to the information processing system 50.

[0258] S25: The communication unit 51 of the information processing system 50 receives a remote meeting creation request, the meeting information acquisition unit 54 obtains a unique meeting ID assigned by the meeting management system 9, and the communication unit 51 transmits the meeting ID to the information recording application 41.

[0259] S26: The conference information acquisition unit 54 also transmits the destination of the combined image video (URL of the storage service system 70) to the information recording application 41 via the communication unit 51.

[0260] S27: The communication unit 11 of the information recording application 41 receives the conference ID and the location where the recording file will be saved, causing the video storage unit 17 to determine that it is ready to record and to start recording.

[0261] S28: The app screen acquisition unit 14 of the information recording app 41 requests the app for the screen of the app selected by the user (more specifically, the app screen acquisition unit 14 acquires the app screen via the OS). In FIG. 23, the app selected by the user is the remote conferencing app 42.

[0262] S29: The video saving unit 17 of the information recording app 41 notifies the meeting device 60 of the start of recording via the device communication unit 16. When notifying, the video saving unit 17 may also notify that the camera toggle button 211 is on (request for panorama image and speaker image). Regardless of the presence or absence of the request, the meeting device 60 transmits the panorama image and the speaker image to the information recording app 41.

[0263] S30: When the device connection unit 67 of the meeting device 60 receives the start of recording, it assigns a non-duplicate recording ID and returns the recording ID to the information recording app 41. Note that the recording ID may be assigned by the information recording app 41 or acquired from the information processing system 50.

[0264] S31: The remote conferencing service system 90 repeatedly transmits the audio data and image data transmitted from other sites to the remote conferencing app.

[0265] S32: The audio acquisition unit 15 of the information recording app 41 acquires the audio data output by the terminal device 10 (the audio data received by the remote conferencing app 42).

[0266] S33: The device communication unit 16 transmits the audio data acquired by the audio acquisition unit 15 and the synthesis request to the meeting device 60.

[0267] S34: The device connection unit 67 of the meeting device 60 receives the audio data and the synthesis request. The sound collection unit 68 always collects the surrounding sound. The site determination processing unit 64 divides the other-site audio B received by the device connection unit 67 and the self-site audio A collected by the sound collection unit 68 according to a predetermined rule, and determines the site based on the sound pressure of each.

[0268] S35: Next, the speech synthesis unit 65 synthesizes the local voice A (surrounding sound data) collected by the sound collection unit 68 and the other-site voice B received by the device connection unit 67. Therefore, the synthesized voice C is generated in a divided state. For example, the speech synthesis unit 65 adds the local voice A and the other-site voice B together. Since clear sound from the vicinity of the meeting device 60 is recorded, the accuracy of text transcription of sound, especially from the vicinity of the meeting device 60 (conference room side), is improved.

[0269] This voice synthesis can also be performed on the terminal device 10. However, by distributing the recording function to the terminal device 10 and the voice processing to the meeting device 60, the load on both the terminal device 10 and the meeting device 60 can be reduced. Alternatively, the recording function may be distributed to the meeting device 60 and the voice processing to the terminal device 10.

[0270] S36: The text conversion request unit 66 of the meeting device 60 transmits a speech recognition request (divided synthesized speech C) and site identification information E to the information processing system 50 via the communication unit 61.

[0271] S37: The communication unit 51 of the information processing system 50 receives a speech recognition request (divided synthesized speech C) and location identification information E. The speech recognition unit 55 transmits the speech recognition request (divided synthesized speech C) to the speech recognition service system and obtains the recognition result string.

[0272] S38: The information processing system 50 returns the recognition result string, voice data, and location identification information E to the information recording application 41. In order for the information processing system 50 to return these to the information recording application 41, the meeting device 60 attaches its own identification information in step S36. In addition, the information recording application 41 has previously set the IP address of the terminal device 10 and the identification information obtained from the meeting device 60 in the information processing system 50. In this way, the information processing system 50 can identify the terminal device 10 based on the identification information of the meeting device 60.

[0273] S39: The location determination result recording unit 58, the speech recognition result recording unit 57, and the speech recording unit 56 of the information processing system 50 save the recognition result string, speech data, and location identification information E to the same storage location as the combined image video via the communication unit 51. A conference ID is attached to these.

[0274] S40: In addition, the panoramic image creation unit 62 of the meeting device 60 creates a panoramic image, and the speaker image creation unit 63 creates a speaker image.

[0275] S41: The device communication unit 16 of the information recording application 41 repeatedly acquires panoramic images and speaker images from the meeting device 60. The device communication unit 16 also repeatedly requests and acquires synthesized audio data from the meeting device 60. These acquisitions may be performed by the device communication unit 16 making requests to the meeting device 60. Alternatively, the meeting device 60 may automatically send the panoramic images and speaker images when it receives a notification that the camera toggle button 211 is ON. The meeting device 60 may automatically send the synthesized audio data to the information recording application 41 when it receives a request to synthesize audio data.

[0276] S42: The display control unit 13 of the information recording application 41 displays the application screen, panoramic image, and speaker image side by side on the recording screen 220. The video storage unit 17 of the information recording application 41 combines the application screen, panoramic image, and speaker image acquired from the remote conferencing application 42 and saves them as a combined image video. In other words, the video storage unit 17 creates a combined image by combining the repeatedly received application screen, panoramic image, and speaker image, and creates a combined image video by specifying the combined image as the frame that makes up the combined image video. The video storage unit 17 also stores the audio data received from the meeting device 60.

[0277] The information recording application 41 repeats the above steps S32 to S42.

[0278] S43: When the remote meeting ends and recording is no longer needed, the user instructs the information recording application 41 to stop recording (for example, by pressing the recording stop button 227). The operation reception unit 12 of the information recording application 41 receives the instruction.

[0279] S44: The device communication unit 16 of the information recording application 41 notifies the meeting device 60 that recording has ended. The meeting device 60 continues to create panoramic images and speaker images and synthesize audio. However, the meeting device 60 may change the processing load, such as changing the resolution or fps, depending on whether recording is in progress.

[0280] S45: The video storage unit 17 of the information recording application 41 combines audio data with the combined image video to create a combined image video with audio. If no panoramic images, speaker images, or application screens are saved, the audio data may be independent.

[0281] S46: The upload unit 20 of the information recording application 41 saves the combined image video to the combined image video storage location via the communication unit 11. In the recording information storage unit 5002, the combined image video is associated with the conference ID and the recording ID. Uploaded information is recorded in the combined image video.

[0282] Since the save location is notified to the user, the user can share the combined image / video with participants by informing them of the save location via email or other means. Even if the devices used to create the combined image / video, audio data, and text data are different, they can be collected and stored in a single location, making them easily viewable by users later.

[0283] Note that the processing in steps S32 to S42 does not have to be in the order shown in Figure 23, and the synthesis of audio data and saving of the combined image video may be performed in any order.

[0284] <<Voice recognition after recording ends>> Next, referring to Fig. 24, the processing procedure for voice recognition after the recording is completed will be described. Fig. 24 is an example of a sequence diagram showing the procedure in which the information recording application 41 records a panoramic image, a speaker image, and the screen of the application. In Fig. 24, mainly the differences from Fig. 23 will be explained. Steps S1 to S35 may be the same as steps S1 to S35 in Fig. 23. However, there are no steps S36 to S39 in which the meeting device 60 requests voice recognition from the information processing system 50.

[0285] S51: The device connection unit 67 of the meeting device 60 transmits a panoramic image, a speaker image, voice data (synthesized voice C), and base identification information E to the information recording application 41. Therefore, at this point, there is no recognition result string, but the voice data is segmented.

[0286] S52: The device communication unit 16 of the information recording application 41 repeatedly acquires a panoramic image, a speaker image, voice data, and base identification information E from the meeting device 60. The display control unit 13 of the information recording application 41 arranges the screen of the application, the panoramic image, and the speaker image and displays them on the recording screen 220.

[0287] S53, S54: The recording end process may be the same as steps S43 and S44 in Fig. 23.

[0288] S55: When the user checks the checkbox 215 associated with "Automatically perform character recognition after uploading the recording" on the recording setting screen 210, the voice data processing unit 18 requests the information processing system 50 to convert the voice data into text data. Specifically, the voice data processing unit 18 specifies the URL of the storage destination and the base identification information E via the communication unit 11, and together with the meeting ID and the recording ID, transmits a conversion request for the voice data (segmented synthesized voice C) combined with the combined image video to the information processing system 50.

[0289] S56: The communication unit 51 of the information processing system 50 receives the conversion request for the voice data, and the voice recognition unit 55 uses the voice recognition service system 80 to convert the voice data into text data.

[0290] S57: The communications unit 51 obtains the recognition result string from the speech recognition service system.

[0291] S58: The base determination result recording unit 58, the speech recognition result recording unit 57, and the speech recording unit 56 save the recognition result string, speech data (divided synthesized speech C), and base identification information E to the same storage location as the combined image video via the communication unit 51.

[0292] S59: The video storage unit 17 of the information recording application 41 combines audio data with the combined image video to create a combined image video with audio. If no panoramic images, speaker images, or application screens are saved, the audio data may be independent.

[0293] S60: The upload unit 20 of the information recording application 41 saves the combined image video to the combined image video storage location via the communication unit 11. In the recording information storage unit 5002, the combined image video is associated with the conference ID and the recording ID. Uploaded information is recorded in the combined image video.

[0294] Thus, even if the information recording app 41 performs voice recognition after recording has finished, it can still determine the location.

[0295] <<Editing location identification information>> Figure 25 is an example of a sequence diagram illustrating the process by which a user edits location identification information E.

[0296] S71: The user connects terminal device 10 to the information processing system 50 and displays the meeting list screen. The meeting list screen displays a list of meetings that the logged-in user has permission to view.

[0297] S72: The user selects a meeting and inputs an operation to display the meeting record confirmation screen 240. The operation reception unit 12 accepts the operation.

[0298] S73: The communication unit 11 of the terminal device 10 requests the information processing system 50 to display the meeting record confirmation screen 240 by specifying the meeting ID.

[0299] S74, S75: The communication unit 51 of the information processing system 50 receives a request for the meeting record confirmation screen 240 and obtains the recognition result string identified by the meeting ID, audio data, and location identification information E from the storage service system 70.

[0300] S76: The meeting record confirmation unit 59 of the information processing system 50 generates a meeting record confirmation screen 240 using the recognized string, audio data, and location identification information E. The meeting record confirmation unit 59 arranges the recognized string on the left and right sides of the meeting record confirmation screen 240 according to the location identification information E.

[0301] S77: The communication unit 51 of the information processing system 50 transmits the screen information of the meeting record confirmation screen 240 to the terminal device 10. The communication unit 11 of the terminal device 10 receives the screen information of the meeting record confirmation screen 240, and the display control unit 13 displays it.

[0302] S78: If a user changes, for example, the location identification information E, the operation reception unit 12 accepts the change.

[0303] S79: The communication unit 11 of the terminal device 10 specifies the conference ID and sends the editing results to the information processing system 50.

[0304] S80: The communication unit 51 of the information processing system 50 receives the editing result, and the location determination result recording unit 58 changes the location identification information E of the text data in the storage service system 70 according to the editing content.

[0305] <Main effects> Thus, the combined video image displays and records a panoramic image of the surroundings including the user, an image of the speaker, and the screen of the application displayed during the remote meeting. Since speech recognition is performed on the synthesized speech C, the processing load on the speech recognition service system can be reduced compared to performing speech recognition separately. In addition, since the location from which the synthesized speech C was spoken is determined based on sound pressure information, it is possible to record which location the speech data was spoken at.

[0306] <Other application examples> Although the best mode for carrying out the present invention has been described above using examples, the present invention is not limited in any way to these examples, and various modifications and substitutions can be made without departing from the spirit of the present invention.

[0307] For example, in this embodiment, only the local voice or voice from another location was determined, but if a location ID is attached to the voice from another location, it is also possible to record which of the other locations the voice is from.

[0308] In this embodiment, the panoramic image, speaker image, and app screen were combined and saved as a combined video; however, these may be saved as separate videos. In this case, when the information recording app 41 plays the video, it places the panoramic image, speaker image, and app screen on the playback screen, respectively.

[0309] Furthermore, the terminal device 10 and the meeting device 60 may be integrated into a single unit. Alternatively, the meeting device 60 may be attached externally to the terminal device 10. Also, the meeting device 60 may consist of a 360-degree camera, microphone, and speaker connected by cables.

[0310] Meeting devices 60 may also be placed at other locations 101. These other locations 101 will separately use meeting devices 60 to create combined image / video and text data. Furthermore, multiple meeting devices 60 may be placed at a single location. In this case, multiple recordings will be created for each meeting device 60.

[0311] Furthermore, the arrangement of the panoramic image 203, speaker image 204, and application screen in the combined image video used in this embodiment is merely an example. The panoramic image 203 may be at the bottom and the speaker image 204 at the top, or the user may change the arrangement, or individually switch the display of the panoramic image 203 and speaker image 204 during playback.

[0312] Furthermore, the configuration examples shown in Figure 8 and other figures are divided according to their main functions to facilitate understanding of the processing performed by the terminal device 10, meeting device 60, and information processing system 50. The present invention is not limited by the way the processing units are divided or the names of those units. The processing of the terminal device 10, meeting device 60, and information processing system 50 can be further divided into more processing units depending on the processing content. Alternatively, a single processing unit can be divided to include even more processing.

[0313] Furthermore, the apparatus described in the examples represents only one of several computing environments for carrying out the embodiments disclosed herein. In one embodiment, the information processing system 50 includes multiple computing devices, such as a server cluster. The multiple computing devices are configured to communicate with each other via any type of communication link, including networks and shared memory, and perform the processing disclosed herein.

[0314] Furthermore, the information processing system 50 can be configured to share the disclosed processing steps, such as those shown in Figure 23, in various combinations. For example, a process executed by a predetermined unit can be executed by multiple information processing devices of the information processing system 50. Also, the information processing system 50 may be consolidated into a single server device or divided into multiple devices.

[0315] Each of the embodiments described above can be implemented by one or more processing circuits. Here, "processing circuit" as used herein includes processors programmed to execute each function by software, such as processors implemented by electronic circuits, as well as devices such as ASICs (Application Specific Integrated Circuits), DSPs (Digital Signal Processors), FPGAs (Field Programmable Gate Arrays), and conventional circuit modules designed to execute each of the functions described above. [Explanation of symbols]

[0316] 10 Terminal devices 50 Information Processing Systems 60 Meeting Devices 100 Record Information Creation System [Prior art documents] [Patent Documents]

[0317] [Patent Document 1] Special Publication No. 2014-206896

Claims

1. A voice processing system in which a terminal device and a device equipped with a microphone communicate, The terminal device receives a first voice from a terminal device at another location via the network. The terminal device includes a speech synthesis unit that synthesizes the first voice received from a terminal device at another location and the second voice collected by the device to generate synthesized voice, A text conversion request unit requests an external party to convert the synthesized speech synthesized by the speech synthesis unit into text data, Before the speech synthesis unit synthesizes the first voice and the second voice, a location determination processing unit determines the location from which the first voice and the second voice were spoken based on the sound pressure of the first voice and the second voice, A voice processing system characterized by having the following features.

2. The aforementioned location determination processing unit divides the first voice and the second voice according to a predetermined rule, Based on the sound pressure of the divided first and second audio signals, the location is determined. The voice processing system according to claim 1, characterized in that, after determining the location, the synthesized voice obtained by combining the first voice and the second voice, the recognition result string obtained by converting the synthesized voice, and the location identification information determined by the location determination processing unit are stored in association with each other.

3. The voice processing system according to claim 1 or 2, characterized in that the location determination processing unit determines that the location from which the greater of the sound pressure of the first voice and the sound pressure of the second voice was spoken is the location from which the synthesized voice created by combining the first voice and the second voice was spoken.

4. The location determination processing unit determines that the synthesized speech was spoken at another location if the sound pressure of the first speech is above a threshold and the sound pressure of the first speech is greater than the sound pressure of the second speech. If the sound pressure of the second voice is above a threshold and is greater than the sound pressure of the first voice, it is determined that the synthesized voice was spoken at the local location. The voice processing system according to claim 1 or 2, characterized in that if the sound pressure of the first voice and the sound pressure of the second voice are below a threshold, it is determined that the location from which the synthesized voice was spoken is unknown.

5. The synthesized voice, the recognition result string obtained by converting the synthesized voice, and the location identification information determined by the location determination processing unit are stored in association on the network. It has an information processing system that communicates with the aforementioned terminal device, The aforementioned information processing system is The conference record confirmation unit provides a terminal device with a screen that displays the recognition result string obtained by converting the synthesized speech and the location identification information determined by the location determination processing unit, in association with each other. The voice processing system according to claim 1, characterized in that the meeting record confirmation unit changes the arrangement of the recognition result string on the screen according to the location identification information.

6. The meeting record verification unit arranges buttons for playing the synthesized speech, associating them with the recognition result string and the location identification information, for each of the divided synthesized speeches. The voice processing system according to claim 5, characterized in that when the aforementioned button is pressed, the terminal device plays the synthesized voice corresponding to the button.

7. The aforementioned terminal device is An operation reception unit that accepts editing of the aforementioned base identification information, The voice processing system according to claim 5 or 6, further comprising a communication unit that transmits the edited location identification information to the information processing system.

8. The terminal device creates recorded information by combining the screen obtained from the application running on the terminal device with the surrounding image obtained by the device. The speech processing system according to any one of claims 1 to 7, characterized in that the synthesized speech is converted into text data by speech recognition and stored in association with the recorded information.

9. The synthesized voice, the recognition result string obtained by converting the synthesized voice, and the location identification information determined by the location determination processing unit are all accompanied by a timestamp. The voice processing system according to claim 2, characterized in that the synthesized voice, the recognition result string, and the location identification information are stored in association with the aforementioned timestamp.

10. A device equipped with a microphone that can communicate with a terminal device, The terminal device receives a first voice from a terminal device at another location via the network. The terminal device includes a speech synthesis unit that synthesizes the first voice received from a terminal device at another location and the second voice collected by the device to generate synthesized voice, A text conversion request unit requests an external party to convert the synthesized speech synthesized by the speech synthesis unit into text data, Before the speech synthesis unit synthesizes the first voice and the second voice, a location determination processing unit determines the location from which the first voice and the second voice were spoken based on the sound pressure of the first voice and the second voice, A device characterized by having the following features.

11. A voice processing method performed by a voice processing system in which a terminal device and a device equipped with a microphone communicate, The terminal device receives a first voice from a terminal device at another location via the network. The speech synthesis unit generates synthesized speech by combining the first voice received by the terminal device from a terminal device at another location with the second voice collected by the device. The text conversion request unit requests an external party to convert the synthesized speech synthesized by the speech synthesis unit into text data, Before the speech synthesis unit synthesizes the first voice and the second voice, the location determination processing unit determines the location from which the first voice and the second voice were spoken based on the sound pressure of the first voice and the second voice, A method for processing sound, characterized by having the following features.

Images