System

Abstract

To provide a system for smoothly proceeding with a meeting among participants who speak different languages.SOLUTION: A system includes a voice recognition section, a translation section, and a voice synthesis section. The voice recognition section performs voice recognition of utterance of a participant in a meeting. The translation section translates utterance recognized by the voice recognition section. The voice synthesis section outputs contents translated by the translation section as voice.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including a description and related instruction sentence regarding the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2022-180282 A Summary of the Invention [Problem to be solved by the invention]

[0004] Conventional techniques have had the problem that it is difficult to smoothly conduct a conference between participants who speak different languages.

[0005] The system according to the embodiment aims to facilitate a conference between participants who speak different languages. [Means for solving the problem]

[0006] The system according to the embodiment includes a speech recognition unit, a translation unit, and a speech synthesis unit. The speech recognition unit recognizes speech of participants in a conference. The translation unit translates the speech recognized by the speech recognition unit. The speech synthesis unit outputs the content translated by the translation unit as speech. Effect of the Invention

[0007] The system according to the embodiment can facilitate conferences between participants who speak different languages. [Brief description of the drawings]

[0008] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Diagram 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. FIG. [Diagram 3] FIG. 11 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Diagram 5] FIG. 13 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 13 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 13 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0010] First, the terms used in the following description will be explained.

[0011] In the following embodiments, a signed processor (hereinafter simply referred to as a "processor") may be one arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be one type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), or a TPU (Tensor Processing Unit).

[0012] In the following embodiments, a signed RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by the processor.

[0013] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0014] In the following embodiments, a communication I / F (Interface) with a code is an interface including a communication processor and an antenna. The communication I / F controls communication between multiple computers. An example of a communication standard applied to the communication I / F is a wireless communication standard including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0015] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. In addition, in this specification, the same idea as "A and / or B" is also applied when three or more things are expressed by connecting them with "and / or."

[0016] [First embodiment] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0017] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0018] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0019] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0020] The reception device 38 includes a touch panel 38A and a microphone 38B, and receives user input. The touch panel 38A detects contact with a pointer (e.g., a pen or a finger) to receive user input by the touch of the pointer. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 (see FIG. 2) acquires the data indicating the user input.

[0021] Output device 40 includes a display 40A and a speaker 40B, and presents data to a user by outputting the data in a form of expression that the user can perceive (e.g., voice and / or text). Display 40A displays visible information such as text and images according to instructions from processor 46. Speaker 40B outputs voice according to instructions from processor 46. Camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a Complementary Metal-Oxide-Semiconductor (CMOS) image sensor or a Charge Coupled Device (CCD) image sensor.

[0022] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for transmitting and receiving various types of information between the processor 46 and the processor 28 via the network 54.

[0023] FIG. 2 shows an example of main functions of the data processing device 12 and the smart device 14.

[0024] As shown in Fig. 2, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32. The specific process program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific process program 56 from the storage 32, and executes the read specific process program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific process program 56 executed on the RAM 30.

[0025] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290. The identification processing unit 290 can estimate the emotion of the user using the emotion identification model 59 and perform identification processing using the user's emotion. In the emotion estimation function (emotion identification function) using the emotion identification model 59, various estimations and predictions regarding the user's emotion, including estimation and prediction of the user's emotion, are performed, but are not limited to such examples. In addition, the estimation and prediction of the emotion also include, for example, analysis of the emotion.

[0026] In the smart device 14, the specific processing is performed by the processor 46. The storage 50 stores a specific processing program 60. The specific processing program 60 is used together with the specific processing program 56 by the data processing system 10. The processor 46 reads the specific processing program 60 from the storage 50 and executes the specific processing program 60 on the RAM 48. The specific processing is realized by the processor 46 operating as the control unit 46A in accordance with the specific processing program 60 executed on the RAM 48. Note that the smart device 14 has a data generation model and an emotion identification model similar to the data generation model 58 and the emotion identification model 59, and can also perform processing similar to that of the specific processing unit 290 using these models.

[0027] Note that a device other than the data processing device 12 may have the data generation model 58. For example, a server device (e.g., a generation server) may have the data generation model 58. In this case, the data processing device 12 obtains a processing result (prediction result, etc.) using the data generation model 58 by communicating with the server device having the data generation model 58. Also, the data processing device 12 may be a server device, or may be a terminal device (e.g., a mobile phone, a robot, a home appliance, etc.) owned by a user. Next, an example of processing by the data processing system 10 according to the first embodiment will be described.

[0028] (Example 1) A conference support system according to an embodiment of the present invention is a system for smoothly conducting a conference between participants who speak different languages. In this conference support system, the contents of what the participants of the conference say in their respective languages ​​are captured by a voice recognition function, translated into another language in real time by a translation function, and output as voice by a voice synthesis function. For example, the speech of a participant who speaks Japanese is converted into text by the voice recognition function, and this text is translated into English by the translation function and conveyed to the English-speaking participant. Furthermore, the translated contents are output as voice by the voice synthesis function. This allows the participant to speak in his / her own language and hear the speech of the participant who speaks another language in his / her own language. For example, the speech of a participant who speaks Japanese is translated into English and output as English voice. In addition, the contents of the conference are automatically recorded as minutes, and the minutes are created by combining the text captured by the voice recognition function and the text generated by the translation function. This allows the contents of the conference to be checked later. Furthermore, the ToDo management function allows the tasks decided in the conference to be managed. For example, the tasks decided during the conference are automatically added to a ToDo list, and the progress can be managed. Finally, the next meeting setup function can automatically set the date and time and participants of the next meeting. This makes it easier to manage the meeting schedule. In this way, the present invention is a system for smoothly conducting a meeting between participants who speak different languages ​​using voice recognition and translation functions, and realizes efficient communication and task management by supporting the creation of minutes, ToDo management, and the setup of the next meeting. In this way, the meeting support system can smoothly conduct a meeting between participants who speak different languages.

[0029] The conference support system according to the embodiment includes a speech recognition unit, a translation unit, and a speech synthesis unit. The speech recognition unit recognizes speech of the conference participants. The speech recognition unit converts speech into text using, for example, an acoustic model or a language model. The speech recognition unit can also apply a speech recognition algorithm using a neural network. For example, the speech recognition unit receives speech data as input, extracts speech features using an acoustic model, and converts the speech into text using a language model. The speech recognition unit can also perform preprocessing of the speech data to improve the accuracy of the speech recognition. For example, preprocessing such as noise removal and volume normalization is performed. The translation unit translates the speech recognized by the speech recognition unit. The translation unit translates the text into another language using, for example, machine translation or neural network translation. The translation unit can perform translation in real time. For example, the translation unit receives text received from the speech recognition unit as input and translates it into another language using a neural network translation model. The translation unit can also take into account context information to improve the accuracy of the translation. For example, the translation unit analyzes the context and selects an appropriate translation. The speech synthesis unit outputs the content translated by the translation unit as speech. The speech synthesis unit converts text into speech using, for example, text to speech synthesis (TTS) or a speech synthesis algorithm. The speech synthesis unit can generate speech in real time. For example, the speech synthesis unit receives text received from the translation unit as input and generates speech using a TTS algorithm. The speech synthesis unit can also adjust the tone and speed of the speech to improve the naturalness of the speech. For example, the speech synthesis unit adjusts the tone and speed of the generated speech to provide a natural speech. As a result, the conference support system according to the embodiment can smoothly conduct a conference between participants who speak different languages.

[0030] The speech recognition unit recognizes speech of participants in a conference. The speech recognition unit converts speech into text using, for example, an acoustic model or a language model. Specifically, the speech recognition unit receives as input speech data collected from multiple microphones installed in a conference room, and breaks it down into speech features using an acoustic model. The acoustic model analyzes the frequency components and temporal changes of the speech to extract basic speech units such as phonemes and syllables. Next, the language model analyzes these speech units and converts them into appropriate words and phrases based on the context. The speech recognition unit can also apply a speech recognition algorithm using a neural network. For example, by using a recurrent neural network (RNN) or a transformer model using deep learning technology, the accuracy of speech recognition can be significantly improved. This enables the speech recognition unit to convert the content of speech into text with high accuracy. Furthermore, the speech recognition unit can also perform preprocessing of the speech data to improve the accuracy of speech recognition. For example, by performing preprocessing such as noise removal and volume normalization, the quality of the speech data can be improved and the recognition accuracy can be increased. Noise removal uses techniques such as spectral subtraction and Wiener filtering, while volume normalization uses peak normalization and RMS normalization. This enables the speech recognition unit to maintain high recognition accuracy even in a variety of environments, supporting smooth conference proceedings.

[0031] The translation unit translates the utterances recognized by the speech recognition unit. The translation unit translates the text into another language, for example, using machine translation or neural network translation. Specifically, the translation unit takes the text received from the speech recognition unit as input and translates it into another language using a neural network translation model. The neural network translation model uses an encoder-decoder architecture, where the encoder converts the input text into an internal representation, and the decoder generates the translated text based on the internal representation. The translation unit can perform translation in real time, so it can provide translation results quickly without disrupting the progress of the meeting. In addition, the translation unit can also take into account contextual information to improve the accuracy of the translation. For example, by analyzing the context before and after and selecting an appropriate translation, a more natural and meaningful translation can be achieved. An attention mechanism using a transformer model is effective for this, and the importance of each part of the input text can be dynamically calculated and reflected in the translation result. In addition, the translation unit can use custom dictionaries and glossaries to handle technical terms and industry-specific expressions. This enables the translation department to handle not only general conversations, but also specialized meetings and technical discussions, facilitating communication between participants.

[0032] The speech synthesis unit outputs the content translated by the translation unit as speech. The speech synthesis unit converts text into speech using, for example, text-to-speech synthesis (TTS) or a speech synthesis algorithm. Specifically, the speech synthesis unit receives the text received from the translation unit as input and generates speech using a TTS algorithm. TTS algorithms include phoneme-based synthesis, concatenative synthesis, and even neural TTS using deep learning. Neural TTS can significantly improve the naturalness and fluency of speech, and the generated speech is very close to human speech. Since the speech synthesis unit can generate speech in real time, it can provide speech output quickly without disrupting the progress of the meeting. In addition, the speech synthesis unit can adjust the tone and speed of the speech to improve the naturalness of the speech. For example, the speech synthesis unit adjusts the tone and speed of the generated speech to provide a natural speech. This uses prosody adjustment technology, which can appropriately control the intonation and rhythm of the speech. In addition, the speech synthesis unit can support multiple voice profiles and simulate the voices of different speakers. This allows the conference participants to select audio output according to their preferences, providing a more comfortable conference experience. As a result, the conference support system according to the embodiment can smoothly conduct a conference between participants who speak different languages.

[0033] The meeting support system includes a minutes-taking unit that takes minutes of a meeting. The minutes-taking unit automatically records the contents of the meeting. The minutes-taking unit takes minutes of a meeting by, for example, combining the text captured by the speech recognition unit and the text generated by the translation unit. The minutes-taking unit can save the minutes so that the contents of the meeting can be checked later. For example, the minutes-taking unit saves the contents of the meeting as a text file so that they can be searched or referenced later. The minutes-taking unit can also summarize the contents of the minutes. For example, the minutes-taking unit extracts important points of the meeting and records them as a summary. This allows the contents of the meeting to be efficiently understood. Some or all of the above-mentioned processing in the minutes-taking unit may be performed, for example, using AI or may be performed without using AI. For example, the minutes-taking unit inputs data acquired from the speech recognition unit and the translation unit into a generation AI, and causes the generation AI to create the minutes. This allows the meeting support system to automatically record the contents of the meeting and check them later.

[0034] The conference support system includes a ToDo management unit that manages ToDos. The ToDo management unit manages tasks decided in a conference. For example, the ToDo management unit automatically adds tasks decided during a conference to a ToDo list. The ToDo management unit can manage the progress of the tasks. For example, the ToDo management unit updates the progress of the tasks in real time and displays the latest information. The ToDo management unit can also set the priority of the tasks. For example, the ToDo management unit sets the priority based on the importance of the tasks and displays important tasks with priority. The ToDo management unit can also manage the completion status of the tasks. For example, the ToDo management unit automatically deletes completed tasks from the list and displays incomplete tasks with priority. This allows the conference support system to manage the tasks decided in the conference. A part or all of the above-mentioned processing in the ToDo management unit may be performed using, for example, AI, or may be performed without using AI. For example, the ToDo management unit can input tasks decided during a conference to a generation AI and cause the generation AI to manage the tasks. This allows the conference support system to efficiently manage the tasks decided in the conference.

[0035] The conference support system includes a conference setup unit that sets up the next conference. The conference setup unit automatically sets the date and time and participants of the next conference. The conference setup unit, for example, selects the optimal date and time in consideration of the schedules of the participants. The conference setup unit can also select the optimal participant by referring to the past participation history of the participants. For example, the conference setup unit selects a date and time when all participants can participate by referring to the calendar information of the participants. The conference setup unit can also select participants based on the theme and agenda of the conference. For example, the conference setup unit selects the optimal participant by referring to the participation history of past conferences. This allows the conference support system to automatically set the date and time and participants of the next conference. A part or all of the above-mentioned processing in the conference setup unit may be performed, for example, using AI, or may be performed without using AI. For example, the conference setup unit inputs schedule information of the participants to the generation AI, and allows the generation AI to select the optimal date and time and participants. This allows the conference support system to efficiently manage the schedule of the next conference.

[0036] The voice recognition unit can convert the speech of the conference participants into text. The voice recognition unit converts speech into text using, for example, an acoustic model or a language model. The voice recognition unit can also apply a voice recognition algorithm using a neural network. For example, the voice recognition unit receives voice data as an input, extracts voice features using an acoustic model, and converts the speech into text using a language model. The voice recognition unit can also perform pre-processing of the voice data to improve the accuracy of the voice recognition. For example, pre-processing such as noise removal and volume normalization is performed. By converting the speech into text, it becomes easier to check the content later. A part or all of the above-mentioned processing in the voice recognition unit may be performed using, for example, AI, or may be performed without using AI. For example, the voice recognition unit can input voice data to a generation AI and have the generation AI execute the text conversion.

[0037] The translation unit can translate the textualized utterance into another language in real time. The translation unit translates the text into another language, for example, using machine translation or neural network translation. The translation unit can perform translation in real time. For example, the translation unit receives text received from the speech recognition unit as input and translates it into another language using a neural network translation model. The translation unit can also take into account context information to improve the accuracy of the translation. For example, the translation unit analyzes the context before and after and selects an appropriate translation. This allows for smooth communication between participants who speak different languages ​​by translating in real time. Some or all of the above-mentioned processing in the translation unit may be performed, for example, using AI or may be performed without using AI. For example, the translation unit can input text data into a generation AI and have the generation AI perform the translation.

[0038] The speech synthesis unit can output the translated content as speech. The speech synthesis unit converts text into speech, for example, using text-to-speech synthesis (TTS) or a speech synthesis algorithm. The speech synthesis unit can generate speech in real time. For example, the speech synthesis unit receives text received from the translation unit as an input and generates speech using a TTS algorithm. The speech synthesis unit can also adjust the tone and speed of the speech to improve the naturalness of the speech. For example, the speech synthesis unit adjusts the tone and speed of the generated speech to provide a natural speech. This allows participants to hear the speech in their own language by outputting the translated content as speech. Some or all of the above-mentioned processing in the speech synthesis unit may be performed, for example, using AI, or may be performed without using AI. For example, the speech synthesis unit can input text data to a generation AI and cause the generation AI to perform speech synthesis.

[0039] The voice recognition unit can remove background sounds of the participant's speech and more clearly recognize the speech content. The voice recognition unit can remove, for example, the sound of an air conditioner in a conference room and clearly recognize the speech content of the participant. The voice recognition unit can also remove keyboard typing sounds generated while the participant is speaking and accurately capture the speech content. The voice recognition unit can also remove external noise while the participant is speaking and clearly recognize the speech content. As a result, the speech content can be clearly recognized by removing the background sound. A part or all of the above-mentioned processing in the voice recognition unit may be performed, for example, using AI or may be performed without using AI. For example, the voice recognition unit can input voice data to a generation AI and cause the generation AI to remove the background sound.

[0040] The voice recognition unit can dynamically adjust the voice recognition parameters according to the speech speed of the participant. For example, when the participant speaks quickly, the voice recognition unit sets the voice recognition parameters to high speed and accurately captures the speech. When the participant speaks slowly, the voice recognition unit can also set the voice recognition parameters to low speed and accurately capture the speech. When the speech speed of the participant varies, the voice recognition unit can dynamically adjust the voice recognition parameters and accurately capture the speech. As a result, the speech can be accurately captured by adjusting the voice recognition parameters according to the speech speed. A part or all of the above-mentioned processing in the voice recognition unit may be performed using, for example, AI, or may be performed without using AI. For example, the voice recognition unit can input voice data to a generation AI and cause the generation AI to execute parameter adjustment according to the speech speed.

[0041] The voice recognition unit can summarize the remarks of the participants in real time and highlight important points. For example, the voice recognition unit can summarize the remarks of the participants in real time and highlight important keywords. The voice recognition unit can also summarize the remarks of the participants in real time and display important points in list form. The voice recognition unit can also summarize the remarks of the participants in real time and display important parts in different colors. This allows the contents of the meeting to be efficiently understood by summarizing the remarks and highlighting important points. Some or all of the above-mentioned processing in the voice recognition unit may be performed, for example, using AI or may be performed without using AI. For example, the voice recognition unit can input voice data to a generation AI and have the generation AI execute summarization and highlighting of important points.

[0042] The voice recognition unit can simultaneously convert the utterances of the participants into text in multiple languages ​​and provide them to participants who speak different languages. For example, the voice recognition unit converts the utterances of the participants into Japanese and English in real time and displays them in both languages. The voice recognition unit can also convert the utterances of the participants into French and German in real time and display them in both languages. The voice recognition unit can also convert the utterances of the participants into Spanish and Chinese in real time and display them in both languages. This makes it easier to communicate between participants who speak different languages ​​by converting the utterances into text in multiple languages. A part or all of the above-mentioned processing in the voice recognition unit may be performed using, for example, AI, or may be performed without using AI. For example, the voice recognition unit can input voice data to a generation AI and have the generation AI execute text conversion in multiple languages.

[0043] The translation unit can dynamically update a dictionary for appropriately translating technical terms and industry-specific words during translation. The translation unit, for example, adds technical terms used during a meeting to the dictionary in real time and translates them appropriately. The translation unit can also detect industry-specific words during a meeting and add them to the dictionary to improve translation accuracy. The translation unit can also add new terms used by participants to the dictionary in real time and translate them appropriately. This improves translation accuracy by appropriately translating technical terms and industry-specific words. Some or all of the above-mentioned processing in the translation unit may be performed, for example, using AI or may be performed without using AI. For example, the translation unit can input technical terms and industry-specific words to a generation AI and cause the generation AI to dynamically update the dictionary.

[0044] The translation unit can apply an algorithm that selects an appropriate translation taking into account the context during translation. The translation unit, for example, selects an appropriate translation taking into account the context before and after the participant's remarks. The translation unit can also select a translation that matches the context based on the theme or agenda of the meeting. The translation unit can also understand the intention of the participant's remarks and select a translation that matches the context. In this way, a more natural translation is provided by selecting a translation taking into account the context. Some or all of the above-mentioned processing in the translation unit may be performed, for example, using AI, or may be performed without using AI. For example, the translation unit can input text data to a generation AI and cause the generation AI to select a translation taking into account the context.

[0045] The translation unit can output the translated content in multiple languages ​​simultaneously and provide it to participants of different languages. For example, the translation unit can translate a Japanese statement into English and French in real time and output it in both languages. The translation unit can also translate an English statement into German and Spanish in real time and output it in both languages. The translation unit can also translate a French statement into Chinese and Japanese in real time and output it in both languages. As a result, by outputting the translated content in multiple languages, communication between participants of different languages ​​becomes smooth. A part or all of the above-mentioned processing in the translation unit may be performed, for example, using AI, or may be performed without using AI. For example, the translation unit can input text data to a generation AI and cause the generation AI to execute translation output in multiple languages.

[0046] The translation unit can summarize the translated content and highlight important points. For example, the translation unit can summarize the translated content and highlight important keywords. The translation unit can also summarize the translated content and display important points in list form. The translation unit can also summarize the translated content and display important parts in different colors. This allows the content of the meeting to be efficiently understood by summarizing the translated content and highlighting important points. Some or all of the above-mentioned processing in the translation unit may be performed, for example, using AI or may be performed without using AI. For example, the translation unit can input text data to a generation AI and have the generation AI execute summarization and highlighting of important points.

[0047] The voice synthesis unit can dynamically adjust parameters for reproducing the characteristics of the speaker's voice during voice synthesis. For example, the voice synthesis unit adjusts the parameters of voice synthesis to reproduce the pitch of the speaker's voice. The voice synthesis unit can also adjust the parameters of voice synthesis to reproduce the speed of the speaker's voice. The voice synthesis unit can also adjust the parameters of voice synthesis to reproduce the intonation of the speaker's voice. This reproduces the characteristics of the speaker's voice, providing a more natural voice. A part or all of the above-mentioned processing in the voice synthesis unit may be performed, for example, using AI, or may be performed without using AI. For example, the voice synthesis unit can input text data to the generation AI and cause the generation AI to adjust parameters for reproducing the characteristics of the voice.

[0048] The voice synthesis unit can appropriately adjust background sounds during voice synthesis to generate easy-to-listen voice. For example, the voice synthesis unit can reduce background sounds during voice synthesis to clarify the content of speech. The voice synthesis unit can also generate natural voice by leaving background sounds appropriately during voice synthesis. The voice synthesis unit can also generate clear voice by completely removing background sounds during voice synthesis. As a result, easy-to-listen voice is provided by appropriately adjusting background sounds. Some or all of the above-mentioned processing in the voice synthesis unit may be performed using AI, for example, or may be performed without using AI. For example, the voice synthesis unit can input text data to the generation AI and cause the generation AI to adjust the background sounds.

[0049] The voice synthesis unit can output the voice-synthesized content in multiple languages ​​simultaneously and provide it to participants of different languages. For example, the voice synthesis unit synthesizes a Japanese utterance into English and French in real time and outputs it in both languages. The voice synthesis unit can also synthesize an English utterance into German and Spanish in real time and output it in both languages. The voice synthesis unit can also synthesize a French utterance into Chinese and Japanese in real time and output it in both languages. As a result, by outputting the voice-synthesized content in multiple languages, communication between participants of different languages ​​becomes smooth. A part or all of the above-mentioned processing in the voice synthesis unit may be performed using, for example, AI, or may be performed without using AI. For example, the voice synthesis unit can input text data to a generation AI and cause the generation AI to execute voice output in multiple languages.

[0050] The voice synthesis unit can summarize the synthesized voice content and emphasize important points. For example, the voice synthesis unit can summarize the synthesized voice content and highlight important keywords. The voice synthesis unit can also summarize the synthesized voice content and display important points in a list format. The voice synthesis unit can also summarize the synthesized voice content and display important parts in different colors. This allows the contents of the meeting to be efficiently understood by summarizing the synthesized voice content and emphasizing important points. A part or all of the above-mentioned processing in the voice synthesis unit may be performed, for example, using AI, or may be performed without using AI. For example, the voice synthesis unit can input text data to a generation AI and have the generation AI execute summarization and emphasis of important points.

[0051] The minutes-taking section can adjust the level of detail based on the importance of the remark when taking minutes. For example, the minutes-taking section records important remarks in detail and other remarks in summary. The minutes-taking section can also add detailed explanations depending on the importance of the remark. For important remarks, the minutes-taking section can also add detailed background information to clarify the speaker's intention. In this way, important remarks can be recorded in detail by adjusting the level of detail based on the importance of the remark. Some or all of the above-mentioned processing in the minutes-taking section may be performed, for example, using AI or may be performed without using AI. For example, the minutes-taking section can input text data to a generation AI and cause the generation AI to adjust the level of detail based on the importance of the remark.

[0052] The minutes-taking unit can generate an appropriate summary by considering the context of the remarks when taking minutes into account. The minutes-taking unit can generate an appropriate summary by considering, for example, the context before and after the remarks. The minutes-taking unit can also generate a summary that matches the context based on the theme or agenda of the meeting. The minutes-taking unit can also understand the intention of the remarks and generate a summary that matches the context. As a result, more accurate minutes are provided by generating summaries by taking into account the context of the remarks. Some or all of the above-mentioned processing in the minutes-taking unit may be performed, for example, using AI, or may be performed without using AI. For example, the minutes-taking unit can input text data to a generation AI and cause the generation AI to generate a summary that takes the context into account.

[0053] The minutes-taking section can simultaneously take minutes in multiple languages ​​and provide them to participants who speak different languages. For example, the minutes-taking section can translate Japanese remarks into English and French in real time and take minutes in both languages. The minutes-taking section can also translate English remarks into German and Spanish in real time and take minutes in both languages. The minutes-taking section can also translate French remarks into Chinese and Japanese in real time and take minutes in both languages. This allows minutes to be taken in multiple languages, facilitating communication between participants who speak different languages. Some or all of the above-mentioned processing in the minutes-taking section can be performed, for example, using AI or without AI. For example, the minutes-taking section can input text data to a generation AI and cause the generation AI to take minutes in multiple languages.

[0054] The minutes-taking section can summarize the minutes and highlight important points. The minutes-taking section can, for example, summarize the minutes and highlight important keywords. The minutes-taking section can also summarize the minutes and display important points in list form. The minutes-taking section can also summarize the minutes and display important parts in different colors. This allows the contents of the meeting to be efficiently understood by summarizing the minutes and highlighting important points. Some or all of the above-mentioned processing in the minutes-taking section can be performed, for example, using AI or without using AI. For example, the minutes-taking section can input text data to a generation AI and have the generation AI execute summarization and highlighting of important points.

[0055] The ToDo management unit can adjust the level of detail of the list based on the importance of the task when managing the ToDo. For example, the ToDo management unit adds detailed explanations to important tasks and displays other tasks in a summarized form. The ToDo management unit can also add detailed background information according to the importance of the task. The ToDo management unit can also display the progress status of important tasks in detail. This allows important tasks to be managed in detail by adjusting the level of detail of the list based on the importance of the task. Some or all of the above-mentioned processing in the ToDo management unit may be performed, for example, using AI or may be performed without using AI. For example, the ToDo management unit can input task data to a generation AI and cause the generation AI to adjust the level of detail based on the importance of the task.

[0056] The ToDo management unit can update the progress of a task in real time during ToDo management. For example, the ToDo management unit updates the progress of a task in real time and displays the latest information. The ToDo management unit can also update the progress of a task in real time and automatically delete a completed task from the list. The ToDo management unit can also update the progress of a task in real time and display an incomplete task preferentially. This allows tasks to be managed based on the latest information by updating the progress of a task in real time. A part or all of the above-mentioned processing in the ToDo management unit may be performed, for example, using AI or may be performed without using AI. For example, the ToDo management unit can input task data to a generation AI and cause the generation AI to execute a real-time update of the progress.

[0057] The ToDo management unit can simultaneously create ToDo lists in multiple languages ​​and provide them to participants of different languages. For example, the ToDo management unit can translate a Japanese task into English and French in real time and create ToDo lists in both languages. The ToDo management unit can also translate an English task into German and Spanish in real time and create ToDo lists in both languages. The ToDo management unit can also translate a French task into Chinese and Japanese in real time and create ToDo lists in both languages. As a result, creating ToDo lists in multiple languages ​​facilitates task management between participants of different languages. A part or all of the above-mentioned processing in the ToDo management unit can be performed, for example, using AI or without using AI. For example, the ToDo management unit can input task data to a generation AI and cause the generation AI to create ToDo lists in multiple languages.

[0058] The ToDo management unit can summarize the ToDo list and emphasize important points. For example, the ToDo management unit can summarize the ToDo list and highlight important keywords. The ToDo management unit can also summarize the ToDo list and display important points in list form. The ToDo management unit can also summarize the ToDo list and display important points in color. This allows the content of the task to be efficiently understood by summarizing the ToDo list and emphasizing important points. Some or all of the above-mentioned processing in the ToDo management unit may be performed, for example, using AI or may be performed without using AI. For example, the ToDo management unit can input task data to a generation AI and have the generation AI execute summarization and emphasis of important points.

[0059] The conference setup unit can select an appropriate date and time in consideration of the schedules of the participants when setting up the conference. The conference setup unit, for example, refers to the calendar information of the participants and selects a date and time when all participants can participate. The conference setup unit can also select the most convenient date and time in consideration of the schedules of the participants. The conference setup unit can also select the optimal date and time in consideration of the past participation history of the participants. In this way, a conference that is easy for all participants to participate in is set by selecting the optimal date and time in consideration of the schedules of the participants. Some or all of the above-mentioned processing in the conference setup unit may be performed, for example, using AI, or may be performed without using AI. For example, the conference setup unit can input schedule data to the generation AI and cause the generation AI to select an appropriate date and time.

[0060] The conference setup unit can select an appropriate participant by referring to the past participation history of the participant when setting up the conference. The conference setup unit, for example, refers to the past participation history of the participant and selects an optimal participant. The conference setup unit can also select an optimal participant by referring to the content of the participant's past remarks. The conference setup unit can also select an optimal participant by referring to the participant's past role. This improves the quality of the conference by selecting an optimal participant by referring to the participant's past participation history. A part or all of the above-mentioned processing in the conference setup unit may be performed, for example, using AI or may be performed without using AI. For example, the conference setup unit can input the participant's history data to the generation AI and cause the generation AI to select an appropriate participant.

[0061] The conference setup unit can simultaneously perform conference setup in multiple languages ​​and provide it to participants of different languages. For example, the conference setup unit can translate a Japanese conference setup into English and French in real time and provide it in both languages. The conference setup unit can also translate an English conference setup into German and Spanish in real time and provide it in both languages. The conference setup unit can also translate a French conference setup into Chinese and Japanese in real time and provide it in both languages. This makes it possible to facilitate communication between participants of different languages ​​by performing the conference setup in multiple languages. A part or all of the above-mentioned processing in the conference setup unit may be performed, for example, using AI or may be performed without using AI. For example, the conference setup unit can input setup data to a generation AI and cause the generation AI to execute a conference setup in multiple languages.

[0062] The conference setup unit can create a summary of the conference setup and highlight important points. For example, the conference setup unit can create a summary of the conference setup and highlight important keywords. The conference setup unit can also create a summary of the conference setup and display important points in a list format. The conference setup unit can also create a summary of the conference setup and display important parts in different colors. This allows the content of the conference to be efficiently understood by creating a summary of the conference setup and highlighting important points. Some or all of the above-mentioned processing in the conference setup unit may be performed, for example, using AI or may be performed without using AI. For example, the conference setup unit can input the setup data to a generation AI and cause the generation AI to execute the summary and highlight the important points.

[0063] The system according to the embodiment is not limited to the above-described example, and various modifications are possible, for example, as follows.

[0064] The conference support system may also include a summarization unit that summarizes what the participants say in real time and highlights important points. The summarization unit, for example, summarizes what the participants say in real time and highlights important keywords. The summarization unit may also summarize what the participants say in real time and display important points in list form. The summarization unit may also summarize what the participants say in real time and display important parts in different colors. In this way, the contents of the conference can be efficiently understood by summarizing what the participants say and highlighting important points.

[0065] The conference support system may also include a multilingual text conversion unit that converts participants' comments into text in multiple languages ​​simultaneously and provides the text to participants who speak different languages. For example, the multilingual text conversion unit converts participants' comments into text in Japanese and English in real time and displays the text in both languages. The multilingual text conversion unit may also convert participants' comments into text in French and German in real time and display the text in both languages. The multilingual text conversion unit may also convert participants' comments into text in Spanish and Chinese in real time and display the text in both languages. In this way, converting comments into text in multiple languages ​​facilitates communication between participants who speak different languages.

[0066] The conference support system may also include a background sound removal unit that removes background sounds of the participants' speech and more clearly recognizes the speech content. The background sound removal unit removes, for example, the sound of an air conditioner in a conference room and clearly recognizes the speech content of the participants. The background sound removal unit can also remove keyboard keystroke sounds generated while a participant is speaking and accurately capture the speech content. The background sound removal unit can also remove external noise while a participant is speaking and clearly recognize the speech content. In this way, by removing the background sound, the speech content can be clearly recognized.

[0067] The conference support system can also include a speed adjustment unit that dynamically adjusts voice recognition parameters according to the speech speed of the participants. For example, when a participant speaks quickly, the speed adjustment unit sets the voice recognition parameters to a high speed to accurately capture the speech. When a participant speaks slowly, the speed adjustment unit can also set the voice recognition parameters to a low speed to accurately capture the speech. When the speech speed of a participant fluctuates, the speed adjustment unit can also dynamically adjust the voice recognition parameters to accurately capture the speech. In this way, speech can be accurately captured by adjusting the voice recognition parameters according to the speech speed.

[0068] The conference support system may also include a dictionary update unit that dynamically updates a dictionary for appropriately translating technical terms and industry-specific words during translation. The dictionary update unit, for example, adds technical terms used during a conference to the dictionary in real time and translates them appropriately. The dictionary update unit may also detect industry-specific words during a conference and add them to the dictionary to improve translation accuracy. The dictionary update unit may also add new terms used by participants to the dictionary in real time and translate them appropriately. This improves translation accuracy by appropriately translating technical terms and industry-specific words.

[0069] The conference support system may also include a multilingual output unit that simultaneously outputs translated content in multiple languages ​​and provides it to participants who speak different languages. For example, the multilingual output unit translates a Japanese comment into English and French in real time and outputs it in both languages. The multilingual output unit may also translate an English comment into German and Spanish in real time and output it in both languages. The multilingual output unit may also translate a French comment into Chinese and Japanese in real time and output it in both languages. In this way, outputting translated content in multiple languages ​​facilitates communication between participants who speak different languages.

[0070] The conference support system may also include a voice reproducing unit that dynamically adjusts parameters for reproducing the characteristics of the speaker's voice during voice synthesis. The voice reproducing unit adjusts the parameters of the voice synthesis to reproduce, for example, the pitch of the speaker's voice. The voice reproducing unit may also adjust the parameters of the voice synthesis to reproduce the speed of the speaker's voice. The voice reproducing unit may also adjust the parameters of the voice synthesis to reproduce the intonation of the speaker's voice. In this way, a more natural voice is provided by reproducing the characteristics of the speaker's voice.

[0071] The process flow of the first embodiment will be briefly described below.

[0072] Step 1: The speech recognition unit recognizes speech of the conference participants. The speech recognition unit converts speech to text using, for example, an acoustic model or a language model. The speech recognition unit can also apply a speech recognition algorithm using a neural network. For example, the speech recognition unit receives speech data as input, extracts speech features using an acoustic model, and converts the data to text using a language model. The speech recognition unit can also perform pre-processing of the speech data to improve the accuracy of speech recognition. For example, pre-processing such as noise removal and volume normalization is performed. Step 2: The translation unit translates the utterance recognized by the speech recognition unit. The translation unit translates the text into another language, for example, using machine translation or neural network translation. The translation unit can perform the translation in real time. For example, the translation unit can take the text received from the speech recognition unit as input and translate it into another language using a neural network translation model. The translation unit can also take into account contextual information to improve the accuracy of the translation, for example, analyzing the context and selecting an appropriate translation. Step 3: The speech synthesis unit outputs the content translated by the translation unit as speech. The speech synthesis unit converts the text into speech, for example, using text-to-speech synthesis (TTS) or a speech synthesis algorithm. The speech synthesis unit can generate speech in real time. For example, the speech synthesis unit takes the text received from the translation unit as input and generates speech using a TTS algorithm. The speech synthesis unit can also adjust the tone and speed of the speech to improve the naturalness of the speech. For example, the speech synthesis unit adjusts the tone and speed of the generated speech to provide a natural speech.

[0073] (Example 2) A conference support system according to an embodiment of the present invention is a system for smoothly conducting a conference between participants who speak different languages. In this conference support system, the contents of what the participants of the conference say in their respective languages ​​are captured by a voice recognition function, translated into another language in real time by a translation function, and output as voice by a voice synthesis function. For example, the speech of a participant who speaks Japanese is converted into text by the voice recognition function, and this text is translated into English by the translation function and conveyed to the English-speaking participant. Furthermore, the translated contents are output as voice by the voice synthesis function. This allows the participant to speak in his / her own language and hear the speech of the participant who speaks another language in his / her own language. For example, the speech of a participant who speaks Japanese is translated into English and output as English voice. In addition, the contents of the conference are automatically recorded as minutes, and the minutes are created by combining the text captured by the voice recognition function and the text generated by the translation function. This allows the contents of the conference to be checked later. Furthermore, the ToDo management function allows the tasks decided in the conference to be managed. For example, the tasks decided during the conference are automatically added to a ToDo list, and the progress can be managed. Finally, the next meeting setup function can automatically set the date and time and participants of the next meeting. This makes it easier to manage the meeting schedule. In this way, the present invention is a system for smoothly conducting a meeting between participants who speak different languages ​​using voice recognition and translation functions, and realizes efficient communication and task management by supporting the creation of minutes, ToDo management, and the setup of the next meeting. In this way, the meeting support system can smoothly conduct a meeting between participants who speak different languages.

[0074] The conference support system according to the embodiment includes a speech recognition unit, a translation unit, and a speech synthesis unit. The speech recognition unit recognizes speech of the conference participants. The speech recognition unit converts speech into text using, for example, an acoustic model or a language model. The speech recognition unit can also apply a speech recognition algorithm using a neural network. For example, the speech recognition unit receives speech data as input, extracts speech features using an acoustic model, and converts the speech into text using a language model. The speech recognition unit can also perform preprocessing of the speech data to improve the accuracy of the speech recognition. For example, preprocessing such as noise removal and volume normalization is performed. The translation unit translates the speech recognized by the speech recognition unit. The translation unit translates the text into another language using, for example, machine translation or neural network translation. The translation unit can perform translation in real time. For example, the translation unit receives text received from the speech recognition unit as input and translates it into another language using a neural network translation model. The translation unit can also take into account context information to improve the accuracy of the translation. For example, the translation unit analyzes the context and selects an appropriate translation. The speech synthesis unit outputs the content translated by the translation unit as speech. The speech synthesis unit converts text into speech using, for example, text to speech synthesis (TTS) or a speech synthesis algorithm. The speech synthesis unit can generate speech in real time. For example, the speech synthesis unit receives text received from the translation unit as input and generates speech using a TTS algorithm. The speech synthesis unit can also adjust the tone and speed of the speech to improve the naturalness of the speech. For example, the speech synthesis unit adjusts the tone and speed of the generated speech to provide a natural speech. As a result, the conference support system according to the embodiment can smoothly conduct a conference between participants who speak different languages.

[0075] The speech recognition unit recognizes speech of participants in a conference. The speech recognition unit converts speech into text using, for example, an acoustic model or a language model. Specifically, the speech recognition unit receives as input speech data collected from multiple microphones installed in a conference room, and breaks it down into speech features using an acoustic model. The acoustic model analyzes the frequency components and temporal changes of the speech to extract basic speech units such as phonemes and syllables. Next, the language model analyzes these speech units and converts them into appropriate words and phrases based on the context. The speech recognition unit can also apply a speech recognition algorithm using a neural network. For example, by using a recurrent neural network (RNN) or a transformer model using deep learning technology, the accuracy of speech recognition can be significantly improved. This enables the speech recognition unit to convert the content of speech into text with high accuracy. Furthermore, the speech recognition unit can also perform preprocessing of the speech data to improve the accuracy of speech recognition. For example, by performing preprocessing such as noise removal and volume normalization, the quality of the speech data can be improved and the recognition accuracy can be increased. Noise removal uses techniques such as spectral subtraction and Wiener filtering, while volume normalization uses peak normalization and RMS normalization. This enables the speech recognition unit to maintain high recognition accuracy even in a variety of environments, supporting smooth conference proceedings.

[0076] The translation unit translates the utterances recognized by the speech recognition unit. The translation unit translates the text into another language, for example, using machine translation or neural network translation. Specifically, the translation unit takes the text received from the speech recognition unit as input and translates it into another language using a neural network translation model. The neural network translation model uses an encoder-decoder architecture, where the encoder converts the input text into an internal representation, and the decoder generates the translated text based on the internal representation. The translation unit can perform translation in real time, so it can provide translation results quickly without disrupting the progress of the meeting. In addition, the translation unit can also take into account contextual information to improve the accuracy of the translation. For example, by analyzing the context before and after and selecting an appropriate translation, a more natural and meaningful translation can be achieved. An attention mechanism using a transformer model is effective for this, and the importance of each part of the input text can be dynamically calculated and reflected in the translation result. In addition, the translation unit can use custom dictionaries and glossaries to handle technical terms and industry-specific expressions. This enables the translation department to handle not only general conversations, but also specialized meetings and technical discussions, facilitating communication between participants.

[0077] The speech synthesis unit outputs the content translated by the translation unit as speech. The speech synthesis unit converts text into speech using, for example, text-to-speech synthesis (TTS) or a speech synthesis algorithm. Specifically, the speech synthesis unit receives the text received from the translation unit as input and generates speech using a TTS algorithm. TTS algorithms include phoneme-based synthesis, concatenative synthesis, and even neural TTS using deep learning. Neural TTS can significantly improve the naturalness and fluency of speech, and the generated speech is very close to human speech. Since the speech synthesis unit can generate speech in real time, it can provide speech output quickly without disrupting the progress of the meeting. In addition, the speech synthesis unit can adjust the tone and speed of the speech to improve the naturalness of the speech. For example, the speech synthesis unit adjusts the tone and speed of the generated speech to provide a natural speech. This uses prosody adjustment technology, which can appropriately control the intonation and rhythm of the speech. In addition, the speech synthesis unit can support multiple voice profiles and simulate the voices of different speakers. This allows the conference participants to select audio output according to their preferences, providing a more comfortable conference experience. As a result, the conference support system according to the embodiment can smoothly conduct a conference between participants who speak different languages.

[0078] The meeting support system includes a minutes-taking unit that takes minutes of a meeting. The minutes-taking unit automatically records the contents of the meeting. The minutes-taking unit takes minutes of a meeting by, for example, combining the text captured by the speech recognition unit and the text generated by the translation unit. The minutes-taking unit can save the minutes so that the contents of the meeting can be checked later. For example, the minutes-taking unit saves the contents of the meeting as a text file so that they can be searched or referenced later. The minutes-taking unit can also summarize the contents of the minutes. For example, the minutes-taking unit extracts important points of the meeting and records them as a summary. This allows the contents of the meeting to be efficiently understood. Some or all of the above-mentioned processing in the minutes-taking unit may be performed, for example, using AI or may be performed without using AI. For example, the minutes-taking unit inputs data acquired from the speech recognition unit and the translation unit into a generation AI, and causes the generation AI to create the minutes. This allows the meeting support system to automatically record the contents of the meeting and check them later.

[0079] The conference support system includes a ToDo management unit that manages ToDos. The ToDo management unit manages tasks decided in a conference. For example, the ToDo management unit automatically adds tasks decided during a conference to a ToDo list. The ToDo management unit can manage the progress of the tasks. For example, the ToDo management unit updates the progress of the tasks in real time and displays the latest information. The ToDo management unit can also set the priority of the tasks. For example, the ToDo management unit sets the priority based on the importance of the tasks and displays important tasks with priority. The ToDo management unit can also manage the completion status of the tasks. For example, the ToDo management unit automatically deletes completed tasks from the list and displays incomplete tasks with priority. This allows the conference support system to manage the tasks decided in the conference. A part or all of the above-mentioned processing in the ToDo management unit may be performed using, for example, AI, or may be performed without using AI. For example, the ToDo management unit can input tasks decided during a conference to a generation AI and cause the generation AI to manage the tasks. This allows the conference support system to efficiently manage the tasks decided in the conference.

[0080] The conference support system includes a conference setup unit that sets up the next conference. The conference setup unit automatically sets the date and time and participants of the next conference. The conference setup unit, for example, selects the optimal date and time in consideration of the schedules of the participants. The conference setup unit can also select the optimal participant by referring to the past participation history of the participants. For example, the conference setup unit selects a date and time when all participants can participate by referring to the calendar information of the participants. The conference setup unit can also select participants based on the theme and agenda of the conference. For example, the conference setup unit selects the optimal participant by referring to the participation history of past conferences. This allows the conference support system to automatically set the date and time and participants of the next conference. A part or all of the above-mentioned processing in the conference setup unit may be performed, for example, using AI, or may be performed without using AI. For example, the conference setup unit inputs schedule information of the participants to the generation AI, and allows the generation AI to select the optimal date and time and participants. This allows the conference support system to efficiently manage the schedule of the next conference.

[0081] The voice recognition unit can convert the speech of the conference participants into text. The voice recognition unit converts speech into text using, for example, an acoustic model or a language model. The voice recognition unit can also apply a voice recognition algorithm using a neural network. For example, the voice recognition unit receives voice data as an input, extracts voice features using an acoustic model, and converts the speech into text using a language model. The voice recognition unit can also perform pre-processing of the voice data to improve the accuracy of the voice recognition. For example, pre-processing such as noise removal and volume normalization is performed. By converting the speech into text, it becomes easier to check the content later. A part or all of the above-mentioned processing in the voice recognition unit may be performed using, for example, AI, or may be performed without using AI. For example, the voice recognition unit can input voice data to a generation AI and have the generation AI execute the text conversion.

[0082] The translation unit can translate the textualized utterance into another language in real time. The translation unit translates the text into another language, for example, using machine translation or neural network translation. The translation unit can perform translation in real time. For example, the translation unit receives text received from the speech recognition unit as input and translates it into another language using a neural network translation model. The translation unit can also take into account context information to improve the accuracy of the translation. For example, the translation unit analyzes the context before and after and selects an appropriate translation. This allows for smooth communication between participants who speak different languages ​​by translating in real time. Some or all of the above-mentioned processing in the translation unit may be performed, for example, using AI or may be performed without using AI. For example, the translation unit can input text data into a generation AI and have the generation AI perform the translation.

[0083] The speech synthesis unit can output the translated content as speech. The speech synthesis unit converts text into speech, for example, using text-to-speech synthesis (TTS) or a speech synthesis algorithm. The speech synthesis unit can generate speech in real time. For example, the speech synthesis unit receives text received from the translation unit as an input and generates speech using a TTS algorithm. The speech synthesis unit can also adjust the tone and speed of the speech to improve the naturalness of the speech. For example, the speech synthesis unit adjusts the tone and speed of the generated speech to provide a natural speech. This allows participants to hear the speech in their own language by outputting the translated content as speech. Some or all of the above-mentioned processing in the speech synthesis unit may be performed, for example, using AI, or may be performed without using AI. For example, the speech synthesis unit can input text data to a generation AI and cause the generation AI to perform speech synthesis.

[0084] The voice recognition unit can estimate the emotion of the participant and adjust the accuracy of the voice recognition based on the estimated emotion. For example, when the participant is nervous, the voice recognition unit increases the sensitivity of the voice recognition to accurately capture the speech. When the participant is relaxed, the voice recognition unit can also set the sensitivity of the voice recognition to normal and capture natural speech. When the participant is excited, the voice recognition unit can also adjust the sensitivity of the voice recognition to remove background sounds and clarify the speech content. In this way, the speech can be accurately captured by adjusting the accuracy of the voice recognition according to the emotion of the participant. The estimation of the emotion is realized using an emotion estimation function using, for example, an emotion engine or a generation AI. The generation AI is, for example, a text generation AI (for example, LLM) or a multimodal generation AI, but is not limited to such examples. A part or all of the above-mentioned processing in the voice recognition unit may be performed using, for example, an AI, or may be performed without using an AI. For example, the voice recognition unit can input the voice data of the participant to the generation AI and cause the generation AI to execute emotion estimation and speech recognition accuracy adjustment.

[0085] The voice recognition unit can remove background sounds of the participant's speech and more clearly recognize the speech content. The voice recognition unit can remove, for example, the sound of an air conditioner in a conference room and clearly recognize the speech content of the participant. The voice recognition unit can also remove keyboard typing sounds generated while the participant is speaking and accurately capture the speech content. The voice recognition unit can also remove external noise while the participant is speaking and clearly recognize the speech content. As a result, the speech content can be clearly recognized by removing the background sound. A part or all of the above-mentioned processing in the voice recognition unit may be performed, for example, using AI or may be performed without using AI. For example, the voice recognition unit can input voice data to a generation AI and cause the generation AI to remove the background sound.

[0086] The voice recognition unit can dynamically adjust the voice recognition parameters according to the speech speed of the participant. For example, when the participant speaks quickly, the voice recognition unit sets the voice recognition parameters to high speed and accurately captures the speech. When the participant speaks slowly, the voice recognition unit can also set the voice recognition parameters to low speed and accurately capture the speech. When the speech speed of the participant varies, the voice recognition unit can dynamically adjust the voice recognition parameters and accurately capture the speech. As a result, the speech can be accurately captured by adjusting the voice recognition parameters according to the speech speed. A part or all of the above-mentioned processing in the voice recognition unit may be performed using, for example, AI, or may be performed without using AI. For example, the voice recognition unit can input voice data to a generation AI and cause the generation AI to execute parameter adjustment according to the speech speed.

[0087] The voice recognition unit can estimate the emotion of the participant and determine the priority of the voice recognition based on the estimated emotion. For example, if the participant is nervous, the voice recognition unit preferentially recognizes the participant's speech and captures it with higher accuracy than other speech. If the participant is relaxed, the voice recognition unit can also recognize all speeches equally and perform balanced capture. If the participant is excited, the voice recognition unit can also preferentially recognize the participant's speech and remove background sounds to capture it clearly. In this way, important speeches can be preferentially recognized by determining the priority of the voice recognition according to the emotion of the participant. The emotion estimation is realized using an emotion estimation function using, for example, an emotion engine or a generation AI. The generation AI is, for example, a text generation AI (for example, LLM) or a multimodal generation AI, but is not limited to such examples. A part or all of the above-mentioned processing in the voice recognition unit may be performed using, for example, an AI, or may be performed without using an AI. For example, the voice recognition unit can input the participant's voice data to the generation AI and cause the generation AI to execute emotion estimation and voice recognition priority determination.

[0088] The voice recognition unit can summarize the remarks of the participants in real time and highlight important points. For example, the voice recognition unit can summarize the remarks of the participants in real time and highlight important keywords. The voice recognition unit can also summarize the remarks of the participants in real time and display important points in list form. The voice recognition unit can also summarize the remarks of the participants in real time and display important parts in different colors. This allows the contents of the meeting to be efficiently understood by summarizing the remarks and highlighting important points. Some or all of the above-mentioned processing in the voice recognition unit may be performed, for example, using AI or may be performed without using AI. For example, the voice recognition unit can input voice data to a generation AI and have the generation AI execute summarization and highlighting of important points.

[0089] The voice recognition unit can simultaneously convert the utterances of the participants into text in multiple languages ​​and provide them to participants who speak different languages. For example, the voice recognition unit converts the utterances of the participants into Japanese and English in real time and displays them in both languages. The voice recognition unit can also convert the utterances of the participants into French and German in real time and display them in both languages. The voice recognition unit can also convert the utterances of the participants into Spanish and Chinese in real time and display them in both languages. This makes it easier to communicate between participants who speak different languages ​​by converting the utterances into text in multiple languages. A part or all of the above-mentioned processing in the voice recognition unit may be performed using, for example, AI, or may be performed without using AI. For example, the voice recognition unit can input voice data to a generation AI and have the generation AI execute text conversion in multiple languages.

[0090] The translation unit can estimate the emotion of the participant and adjust the expression of the translation based on the estimated emotion. For example, if the participant is nervous, the translation unit adjusts the expression of the translation to be polite and calm. If the participant is relaxed, the translation unit can also adjust the expression of the translation to be casual and friendly. If the participant is excited, the translation unit can also adjust the expression of the translation to be energetic and lively. In this way, a more appropriate translation is provided by adjusting the expression of the translation according to the emotion of the participant. The estimation of the emotion is realized using an emotion estimation function using, for example, an emotion engine or a generation AI. The generation AI is a text generation AI (e.g., LLM) or a multimodal generation AI, but is not limited to such examples. A part or all of the above-mentioned processing in the translation unit may be performed, for example, using an AI, or may be performed without using an AI. For example, the translation unit can input text data to the generation AI and cause the generation AI to estimate the emotion and adjust the expression of the translation.

[0091] The translation unit can dynamically update a dictionary for appropriately translating technical terms and industry-specific words during translation. The translation unit, for example, adds technical terms used during a meeting to the dictionary in real time and translates them appropriately. The translation unit can also detect industry-specific words during a meeting and add them to the dictionary to improve translation accuracy. The translation unit can also add new terms used by participants to the dictionary in real time and translate them appropriately. This improves translation accuracy by appropriately translating technical terms and industry-specific words. Some or all of the above-mentioned processing in the translation unit may be performed, for example, using AI or may be performed without using AI. For example, the translation unit can input technical terms and industry-specific words to a generation AI and cause the generation AI to dynamically update the dictionary.

[0092] The translation unit can apply an algorithm that selects an appropriate translation taking into account the context during translation. The translation unit, for example, selects an appropriate translation taking into account the context before and after the participant's remarks. The translation unit can also select a translation that matches the context based on the theme or agenda of the meeting. The translation unit can also understand the intention of the participant's remarks and select a translation that matches the context. In this way, a more natural translation is provided by selecting a translation taking into account the context. Some or all of the above-mentioned processing in the translation unit may be performed, for example, using AI, or may be performed without using AI. For example, the translation unit can input text data to a generation AI and cause the generation AI to select a translation taking into account the context.

[0093] The translation unit can estimate the emotion of the participant and determine the priority of the translation based on the estimated emotion. For example, if the participant is nervous, the translation unit prioritizes the translation of the participant's remark and provides it quickly. If the participant is relaxed, the translation unit can translate all remarks equally and provide a balanced translation. If the participant is excited, the translation unit can prioritize the translation of the participant's remark and provide it clearly by removing background sounds. In this way, important remarks can be translated preferentially by determining the priority of the translation according to the emotion of the participant. The estimation of emotion is realized using an emotion estimation function using, for example, an emotion engine or a generation AI. The generation AI is, for example, a text generation AI (for example, LLM) or a multimodal generation AI, but is not limited to such examples. A part or all of the above-mentioned processing in the translation unit may be performed using, for example, an AI, or may be performed without using an AI. For example, the translation unit can input text data to the generation AI and cause the generation AI to execute emotion estimation and translation priority determination.

[0094] The translation unit can output the translated content in multiple languages ​​simultaneously and provide it to participants of different languages. For example, the translation unit can translate a Japanese statement into English and French in real time and output it in both languages. The translation unit can also translate an English statement into German and Spanish in real time and output it in both languages. The translation unit can also translate a French statement into Chinese and Japanese in real time and output it in both languages. As a result, by outputting the translated content in multiple languages, communication between participants of different languages ​​becomes smooth. A part or all of the above-mentioned processing in the translation unit may be performed, for example, using AI, or may be performed without using AI. For example, the translation unit can input text data to a generation AI and cause the generation AI to execute translation output in multiple languages.

[0095] The translation unit can summarize the translated content and highlight important points. For example, the translation unit can summarize the translated content and highlight important keywords. The translation unit can also summarize the translated content and display important points in list form. The translation unit can also summarize the translated content and display important parts in different colors. This allows the content of the meeting to be efficiently understood by summarizing the translated content and highlighting important points. Some or all of the above-mentioned processing in the translation unit may be performed, for example, using AI or may be performed without using AI. For example, the translation unit can input text data to a generation AI and have the generation AI execute summarization and highlighting of important points.

[0096] The voice synthesis unit can estimate the emotion of the participant and adjust the tone and speed of the voice based on the estimated emotion. For example, if the participant is nervous, the voice synthesis unit synthesizes the voice with a calm tone. If the participant is relaxed, the voice synthesis unit can also synthesize the voice with a bright tone. If the participant is excited, the voice synthesis unit can also synthesize the voice with an energetic tone. This provides a more natural voice by adjusting the tone and speed of the voice according to the emotion of the participant. The emotion estimation is realized using an emotion estimation function using, for example, an emotion engine or a generation AI. The generation AI is a text generation AI (for example, LLM) or a multimodal generation AI, but is not limited to such examples. A part or all of the above-mentioned processing in the voice synthesis unit may be performed using, for example, an AI, or may be performed without using an AI. For example, the voice synthesis unit can input text data to the generation AI and cause the generation AI to estimate the emotion and adjust the tone and speed of the voice.

[0097] The voice synthesis unit can dynamically adjust parameters for reproducing the characteristics of the speaker's voice during voice synthesis. For example, the voice synthesis unit adjusts the parameters of voice synthesis to reproduce the pitch of the speaker's voice. The voice synthesis unit can also adjust the parameters of voice synthesis to reproduce the speed of the speaker's voice. The voice synthesis unit can also adjust the parameters of voice synthesis to reproduce the intonation of the speaker's voice. This reproduces the characteristics of the speaker's voice, providing a more natural voice. A part or all of the above-mentioned processing in the voice synthesis unit may be performed, for example, using AI, or may be performed without using AI. For example, the voice synthesis unit can input text data to the generation AI and cause the generation AI to adjust parameters for reproducing the characteristics of the voice.

[0098] The voice synthesis unit can appropriately adjust background sounds during voice synthesis to generate easy-to-listen voice. For example, the voice synthesis unit can reduce background sounds during voice synthesis to clarify the content of speech. The voice synthesis unit can also generate natural voice by leaving background sounds appropriately during voice synthesis. The voice synthesis unit can also generate clear voice by completely removing background sounds during voice synthesis. As a result, easy-to-listen voice is provided by appropriately adjusting background sounds. Some or all of the above-mentioned processing in the voice synthesis unit may be performed using AI, for example, or may be performed without using AI. For example, the voice synthesis unit can input text data to the generation AI and cause the generation AI to adjust the background sounds.

[0099] The voice synthesis unit can estimate the emotion of the participant and determine the priority of the voice based on the estimated emotion. For example, when a participant is nervous, the voice synthesis unit prioritizes the voice synthesis of the participant's utterance and provides it quickly. When a participant is relaxed, the voice synthesis unit can also synthesize all utterances equally and provide a balanced presentation. When a participant is excited, the voice synthesis unit can also prioritize the voice synthesis of the participant's utterance and provide it clearly by removing background sounds. In this way, important utterances can be preferentially synthesized by determining the priority of the voice according to the emotion of the participant. The estimation of the emotion is realized using an emotion estimation function using, for example, an emotion engine or a generation AI. The generation AI is, for example, a text generation AI (for example, LLM) or a multimodal generation AI, but is not limited to such examples. A part or all of the above-mentioned processing in the voice synthesis unit may be performed using, for example, an AI, or may be performed without using an AI. For example, the voice synthesis unit can input text data to the generation AI and cause the generation AI to execute emotion estimation and voice priority determination.

[0100] The voice synthesis unit can output the voice-synthesized content in multiple languages ​​simultaneously and provide it to participants of different languages. For example, the voice synthesis unit synthesizes a Japanese utterance into English and French in real time and outputs it in both languages. The voice synthesis unit can also synthesize an English utterance into German and Spanish in real time and output it in both languages. The voice synthesis unit can also synthesize a French utterance into Chinese and Japanese in real time and output it in both languages. As a result, by outputting the voice-synthesized content in multiple languages, communication between participants of different languages ​​becomes smooth. A part or all of the above-mentioned processing in the voice synthesis unit may be performed using, for example, AI, or may be performed without using AI. For example, the voice synthesis unit can input text data to a generation AI and cause the generation AI to execute voice output in multiple languages.

[0101] The voice synthesis unit can summarize the synthesized voice content and emphasize important points. For example, the voice synthesis unit can summarize the synthesized voice content and highlight important keywords. The voice synthesis unit can also summarize the synthesized voice content and display important points in a list format. The voice synthesis unit can also summarize the synthesized voice content and display important parts in different colors. This allows the contents of the meeting to be efficiently understood by summarizing the synthesized voice content and emphasizing important points. A part or all of the above-mentioned processing in the voice synthesis unit may be performed, for example, using AI, or may be performed without using AI. For example, the voice synthesis unit can input text data to a generation AI and have the generation AI execute summarization and emphasis of important points.

[0102] The minutes creation unit can estimate the emotions of the participants and adjust the expression of the minutes based on the estimated emotions. For example, if the participants are nervous, the minutes creation unit adjusts the expression of the minutes to be polite and calm. If the participants are relaxed, the minutes creation unit can also adjust the expression of the minutes to be casual and friendly. If the participants are excited, the minutes creation unit can also adjust the expression of the minutes to be energetic and lively. In this way, by adjusting the expression of the minutes according to the emotions of the participants, more appropriate minutes are provided. The estimation of emotions is realized using an emotion estimation function using, for example, an emotion engine or a generation AI. The generation AI is, for example, a text generation AI (for example, LLM) or a multimodal generation AI, but is not limited to such examples. A part or all of the above-mentioned processing in the minutes creation unit may be performed, for example, using an AI, or may be performed without using an AI. For example, the minutes-creation department can input text data into a generation AI and have the generation AI estimate emotions and adjust the way the minutes are expressed.

[0103] The minutes-taking section can adjust the level of detail based on the importance of the remark when taking minutes. For example, the minutes-taking section records important remarks in detail and other remarks in summary. The minutes-taking section can also add detailed explanations depending on the importance of the remark. For important remarks, the minutes-taking section can also add detailed background information to clarify the speaker's intention. In this way, important remarks can be recorded in detail by adjusting the level of detail based on the importance of the remark. Some or all of the above-mentioned processing in the minutes-taking section may be performed, for example, using AI or may be performed without using AI. For example, the minutes-taking section can input text data to a generation AI and cause the generation AI to adjust the level of detail based on the importance of the remark.

[0104] The minutes-taking unit can generate an appropriate summary by considering the context of the remarks when taking minutes into account. The minutes-taking unit can generate an appropriate summary by considering, for example, the context before and after the remarks. The minutes-taking unit can also generate a summary that matches the context based on the theme or agenda of the meeting. The minutes-taking unit can also understand the intention of the remarks and generate a summary that matches the context. As a result, more accurate minutes are provided by generating summaries by taking into account the context of the remarks. Some or all of the above-mentioned processing in the minutes-taking unit may be performed, for example, using AI, or may be performed without using AI. For example, the minutes-taking unit can input text data to a generation AI and cause the generation AI to generate a summary that takes the context into account.

[0105] The minutes-taking unit can estimate the emotions of the participants and determine the priority of the minutes based on the estimated emotions. For example, when a participant is nervous, the minutes-taking unit creates the minutes of the remarks of the participant with priority and provides them quickly. When a participant is relaxed, the minutes-taking unit can record all remarks equally in the minutes to provide a balanced presentation. When a participant is excited, the minutes-taking unit can create the minutes of the remarks with priority and provide them clearly by removing background sounds. This allows important remarks to be recorded with priority by determining the priority of the minutes according to the emotions of the participants. The estimation of emotions is realized using an emotion estimation function using, for example, an emotion engine or a generation AI. The generation AI is, for example, a text generation AI (for example, LLM) or a multimodal generation AI, but is not limited to such examples. A part or all of the above-mentioned processing in the minutes-taking unit may be performed, for example, using an AI, or may be performed without using an AI. For example, the minutes-creation department can input text data into a generation AI and have the generation AI estimate emotions and determine priorities for the minutes.

[0106] The minutes-taking section can simultaneously take minutes in multiple languages ​​and provide them to participants who speak different languages. For example, the minutes-taking section can translate Japanese remarks into English and French in real time and take minutes in both languages. The minutes-taking section can also translate English remarks into German and Spanish in real time and take minutes in both languages. The minutes-taking section can also translate French remarks into Chinese and Japanese in real time and take minutes in both languages. This allows minutes to be taken in multiple languages, facilitating communication between participants who speak different languages. Some or all of the above-mentioned processing in the minutes-taking section can be performed, for example, using AI or without AI. For example, the minutes-taking section can input text data to a generation AI and cause the generation AI to take minutes in multiple languages.

[0107] The minutes-taking section can summarize the minutes and highlight important points. The minutes-taking section can, for example, summarize the minutes and highlight important keywords. The minutes-taking section can also summarize the minutes and display important points in list form. The minutes-taking section can also summarize the minutes and display important parts in different colors. This allows the contents of the meeting to be efficiently understood by summarizing the minutes and highlighting important points. Some or all of the above-mentioned processing in the minutes-taking section can be performed, for example, using AI or without using AI. For example, the minutes-taking section can input text data to a generation AI and have the generation AI execute summarization and highlighting of important points.

[0108] The ToDo management unit can estimate the emotions of the participants and adjust the priority of the ToDo list based on the estimated emotions. For example, when a participant is nervous, the ToDo management unit can display important tasks with priority so that the participant can respond quickly. When a participant is relaxed, the ToDo management unit can also display all tasks equally to perform balanced management. When a participant is excited, the ToDo management unit can also display important tasks with priority and add background information to clarify. This allows important tasks to be managed with priority by adjusting the priority of the ToDo list according to the participant's emotions. The estimation of emotions is realized using an emotion estimation function using, for example, an emotion engine or a generation AI. The generation AI is, for example, a text generation AI (for example, LLM) or a multimodal generation AI, but is not limited to such examples. A part or all of the above-mentioned processing in the ToDo management unit may be performed using, for example, an AI, or may be performed without using an AI. For example, the ToDo management unit can input task data to the generation AI and cause the generation AI to estimate emotions and adjust the priority of the ToDo list.

[0109] The ToDo management unit can adjust the level of detail of the list based on the importance of the task when managing the ToDo. For example, the ToDo management unit adds detailed explanations to important tasks and displays other tasks in a summarized form. The ToDo management unit can also add detailed background information according to the importance of the task. The ToDo management unit can also display the progress status of important tasks in detail. This allows important tasks to be managed in detail by adjusting the level of detail of the list based on the importance of the task. Some or all of the above-mentioned processing in the ToDo management unit may be performed, for example, using AI or may be performed without using AI. For example, the ToDo management unit can input task data to a generation AI and cause the generation AI to adjust the level of detail based on the importance of the task.

[0110] The ToDo management unit can update the progress of a task in real time during ToDo management. For example, the ToDo management unit updates the progress of a task in real time and displays the latest information. The ToDo management unit can also update the progress of a task in real time and automatically delete a completed task from the list. The ToDo management unit can also update the progress of a task in real time and display an incomplete task preferentially. This allows tasks to be managed based on the latest information by updating the progress of a task in real time. A part or all of the above-mentioned processing in the ToDo management unit may be performed, for example, using AI or may be performed without using AI. For example, the ToDo management unit can input task data to a generation AI and cause the generation AI to execute a real-time update of the progress.

[0111] The ToDo management unit can estimate the emotions of the participants and adjust the display method of the ToDo list based on the estimated emotions. For example, when a participant is nervous, the ToDo management unit provides a simple and highly visible display method. When a participant is relaxed, the ToDo management unit can also provide a display method including detailed information. When a participant is excited, the ToDo management unit can also provide a visually stimulating display method. This enables more appropriate task management by adjusting the display method of the ToDo list according to the emotions of the participants. The estimation of emotions is realized using an emotion estimation function using, for example, an emotion engine or a generation AI. The generation AI is, for example, a text generation AI (for example, LLM) or a multimodal generation AI, but is not limited to such examples. A part or all of the above-mentioned processing in the ToDo management unit may be performed using, for example, an AI, or may be performed without using an AI. For example, the ToDo management unit can input task data to the generation AI and cause the generation AI to estimate emotions and adjust the display method of the ToDo list.

[0112] The ToDo management unit can simultaneously create ToDo lists in multiple languages ​​and provide them to participants of different languages. For example, the ToDo management unit can translate a Japanese task into English and French in real time and create ToDo lists in both languages. The ToDo management unit can also translate an English task into German and Spanish in real time and create ToDo lists in both languages. The ToDo management unit can also translate a French task into Chinese and Japanese in real time and create ToDo lists in both languages. As a result, creating ToDo lists in multiple languages ​​facilitates task management between participants of different languages. A part or all of the above-mentioned processing in the ToDo management unit can be performed, for example, using AI or without using AI. For example, the ToDo management unit can input task data to a generation AI and cause the generation AI to create ToDo lists in multiple languages.

[0113] The ToDo management unit can summarize the ToDo list and emphasize important points. For example, the ToDo management unit can summarize the ToDo list and highlight important keywords. The ToDo management unit can also summarize the ToDo list and display important points in list form. The ToDo management unit can also summarize the ToDo list and display important points in color. This allows the content of the task to be efficiently understood by summarizing the ToDo list and emphasizing important points. Some or all of the above-mentioned processing in the ToDo management unit may be performed, for example, using AI or may be performed without using AI. For example, the ToDo management unit can input task data to a generation AI and have the generation AI execute summarization and emphasis of important points.

[0114] The conference setup unit can estimate the emotions of the participants and adjust the date and time of the next conference based on the estimated emotions. For example, if the participants are nervous, the conference setup unit sets the next conference to a time period when the participants can relax. If the participants are relaxed, the conference setup unit can also set the next conference to a time period when all participants can easily participate. If the participants are excited, the conference setup unit can also set the next conference to an earlier time period. This allows a more appropriate conference schedule to be set by adjusting the date and time of the next conference according to the emotions of the participants. The estimation of emotions is realized using an emotion estimation function using, for example, an emotion engine or a generation AI. The generation AI is, for example, a text generation AI (for example, LLM) or a multimodal generation AI, but is not limited to such examples. A part or all of the above-mentioned processing in the conference setup unit may be performed using, for example, an AI, or may be performed without using an AI. For example, the conference setup unit can input schedule data to the generation AI and cause the generation AI to execute emotion estimation and adjustment of the conference date and time.

[0115] The conference setup unit can select an appropriate date and time in consideration of the schedules of the participants when setting up the conference. The conference setup unit, for example, refers to the calendar information of the participants and selects a date and time when all participants can participate. The conference setup unit can also select the most convenient date and time in consideration of the schedules of the participants. The conference setup unit can also select the optimal date and time in consideration of the past participation history of the participants. In this way, a conference that is easy for all participants to participate in is set by selecting the optimal date and time in consideration of the schedules of the participants. Some or all of the above-mentioned processing in the conference setup unit may be performed, for example, using AI, or may be performed without using AI. For example, the conference setup unit can input schedule data to the generation AI and cause the generation AI to select an appropriate date and time.

[0116] The conference setup unit can select an appropriate participant by referring to the past participation history of the participant when setting up the conference. The conference setup unit, for example, refers to the past participation history of the participant and selects an optimal participant. The conference setup unit can also select an optimal participant by referring to the content of the participant's past remarks. The conference setup unit can also select an optimal participant by referring to the participant's past role. This improves the quality of the conference by selecting an optimal participant by referring to the participant's past participation history. A part or all of the above-mentioned processing in the conference setup unit may be performed, for example, using AI or may be performed without using AI. For example, the conference setup unit can input the participant's history data to the generation AI and cause the generation AI to select an appropriate participant.

[0117] The conference setup unit can estimate the emotions of the participants and determine the priority of the conference based on the estimated emotions. For example, if a participant is nervous, the conference setup unit prioritizes the conference and responds quickly. If a participant is relaxed, the conference setup unit can also set all conferences equally and respond in a balanced manner. If a participant is excited, the conference setup unit can also prioritize the conference and add background information to clarify. In this way, important conferences can be prioritized by determining the priority of the conference according to the emotions of the participants. The estimation of emotions is realized using an emotion estimation function using, for example, an emotion engine or a generation AI. The generation AI is, for example, a text generation AI (for example, LLM) or a multimodal generation AI, but is not limited to such examples. A part or all of the above-mentioned processing in the conference setup unit may be performed using, for example, an AI, or may be performed without using an AI. For example, the conference setup unit can input schedule data to the generation AI and cause the generation AI to execute emotion estimation and conference priority determination.

[0118] The conference setup unit can simultaneously perform conference setup in multiple languages ​​and provide it to participants of different languages. For example, the conference setup unit can translate a Japanese conference setup into English and French in real time and provide it in both languages. The conference setup unit can also translate an English conference setup into German and Spanish in real time and provide it in both languages. The conference setup unit can also translate a French conference setup into Chinese and Japanese in real time and provide it in both languages. This makes it possible to facilitate communication between participants of different languages ​​by performing the conference setup in multiple languages. A part or all of the above-mentioned processing in the conference setup unit may be performed, for example, using AI or may be performed without using AI. For example, the conference setup unit can input setup data to a generation AI and cause the generation AI to execute a conference setup in multiple languages.

[0119] The conference setup unit can create a summary of the conference setup and highlight important points. For example, the conference setup unit can create a summary of the conference setup and highlight important keywords. The conference setup unit can also create a summary of the conference setup and display important points in a list format. The conference setup unit can also create a summary of the conference setup and display important parts in different colors. This allows the content of the conference to be efficiently understood by creating a summary of the conference setup and highlighting important points. Some or all of the above-mentioned processing in the conference setup unit may be performed, for example, using AI or may be performed without using AI. For example, the conference setup unit can input the setup data to a generation AI and cause the generation AI to execute the summary and highlight the important points.

[0120] The system according to the embodiment is not limited to the above-described example, and various modifications are possible, for example, as follows.

[0121] The conference support system may also include a summarization unit that summarizes what the participants say in real time and highlights important points. The summarization unit, for example, summarizes what the participants say in real time and highlights important keywords. The summarization unit may also summarize what the participants say in real time and display important points in list form. The summarization unit may also summarize what the participants say in real time and display important parts in different colors. In this way, the contents of the conference can be efficiently understood by summarizing what the participants say and highlighting important points.

[0122] The conference support system may also include a multilingual text conversion unit that converts participants' comments into text in multiple languages ​​simultaneously and provides the text to participants who speak different languages. For example, the multilingual text conversion unit converts participants' comments into text in Japanese and English in real time and displays the text in both languages. The multilingual text conversion unit may also convert participants' comments into text in French and German in real time and display the text in both languages. The multilingual text conversion unit may also convert participants' comments into text in Spanish and Chinese in real time and display the text in both languages. In this way, converting comments into text in multiple languages ​​facilitates communication between participants who speak different languages.

[0123] The conference support system may also include an emotion adjustment unit that estimates the emotion of a participant and adjusts the accuracy of voice recognition based on the estimated emotion. For example, when a participant is nervous, the emotion adjustment unit increases the sensitivity of voice recognition to accurately capture speech. When a participant is relaxed, the emotion adjustment unit can also set the sensitivity of voice recognition to normal and capture natural speech. When a participant is excited, the emotion adjustment unit can also adjust the sensitivity of voice recognition to remove background sounds and clarify the content of speech. In this way, speech can be accurately captured by adjusting the accuracy of voice recognition according to the emotion of the participant.

[0124] The conference support system may also include a background sound removal unit that removes background sounds of the participants' speech and more clearly recognizes the speech content. The background sound removal unit removes, for example, the sound of an air conditioner in a conference room and clearly recognizes the speech content of the participants. The background sound removal unit can also remove keyboard keystroke sounds generated while a participant is speaking and accurately capture the speech content. The background sound removal unit can also remove external noise while a participant is speaking and clearly recognize the speech content. In this way, by removing the background sound, the speech content can be clearly recognized.

[0125] The conference support system can also include a speed adjustment unit that dynamically adjusts voice recognition parameters according to the speech speed of the participants. For example, when a participant speaks quickly, the speed adjustment unit sets the voice recognition parameters to a high speed to accurately capture the speech. When a participant speaks slowly, the speed adjustment unit can also set the voice recognition parameters to a low speed to accurately capture the speech. When the speech speed of a participant fluctuates, the speed adjustment unit can also dynamically adjust the voice recognition parameters to accurately capture the speech. In this way, speech can be accurately captured by adjusting the voice recognition parameters according to the speech speed.

[0126] The conference support system may also include an emotion translation unit that estimates the emotion of a participant and adjusts the translation expression based on the estimated emotion. For example, if a participant is nervous, the emotion translation unit adjusts the translation expression to be polite and calm. If a participant is relaxed, the emotion translation unit may also adjust the translation expression to be casual and friendly. If a participant is excited, the emotion translation unit may also adjust the translation expression to be energetic and lively. In this way, a more appropriate translation is provided by adjusting the translation expression according to the emotion of the participant.

[0127] The conference support system may also include a dictionary update unit that dynamically updates a dictionary for appropriately translating technical terms and industry-specific words during translation. The dictionary update unit, for example, adds technical terms used during a conference to the dictionary in real time and translates them appropriately. The dictionary update unit may also detect industry-specific words during a conference and add them to the dictionary to improve translation accuracy. The dictionary update unit may also add new terms used by participants to the dictionary in real time and translate them appropriately. This improves translation accuracy by appropriately translating technical terms and industry-specific words.

[0128] The conference support system may also include a multilingual output unit that simultaneously outputs translated content in multiple languages ​​and provides it to participants who speak different languages. For example, the multilingual output unit translates a Japanese comment into English and French in real time and outputs it in both languages. The multilingual output unit may also translate an English comment into German and Spanish in real time and output it in both languages. The multilingual output unit may also translate a French comment into Chinese and Japanese in real time and output it in both languages. In this way, outputting translated content in multiple languages ​​facilitates communication between participants who speak different languages.

[0129] The conference support system may also include an emotional voice adjustment unit that estimates the emotion of a participant and adjusts the tone and speed of the voice based on the estimated emotion. For example, the emotional voice adjustment unit synthesizes a voice with a calm tone when a participant is nervous. The emotional voice adjustment unit may also synthesize a voice with a bright tone when a participant is relaxed. The emotional voice adjustment unit may also synthesize a voice with an energetic tone when a participant is excited. In this way, a more natural voice is provided by adjusting the tone and speed of the voice according to the emotion of the participant.

[0130] The conference support system may also include a voice reproducing unit that dynamically adjusts parameters for reproducing the characteristics of the speaker's voice during voice synthesis. The voice reproducing unit adjusts the parameters of the voice synthesis to reproduce, for example, the pitch of the speaker's voice. The voice reproducing unit may also adjust the parameters of the voice synthesis to reproduce the speed of the speaker's voice. The voice reproducing unit may also adjust the parameters of the voice synthesis to reproduce the intonation of the speaker's voice. In this way, a more natural voice is provided by reproducing the characteristics of the speaker's voice.

[0131] The process flow of the second embodiment will be briefly described below.

[0132] Step 1: The speech recognition unit recognizes speech of the conference participants. The speech recognition unit converts speech to text using, for example, an acoustic model or a language model. The speech recognition unit can also apply a speech recognition algorithm using a neural network. For example, the speech recognition unit receives speech data as input, extracts speech features using an acoustic model, and converts the data to text using a language model. The speech recognition unit can also perform pre-processing of the speech data to improve the accuracy of speech recognition. For example, pre-processing such as noise removal and volume normalization is performed. Step 2: The translation unit translates the utterance recognized by the speech recognition unit. The translation unit translates the text into another language, for example, using machine translation or neural network translation. The translation unit can perform the translation in real time. For example, the translation unit can take the text received from the speech recognition unit as input and translate it into another language using a neural network translation model. The translation unit can also take into account contextual information to improve the accuracy of the translation, for example, analyzing the context and selecting an appropriate translation. Step 3: The speech synthesis unit outputs the content translated by the translation unit as speech. The speech synthesis unit converts the text into speech, for example, using text-to-speech synthesis (TTS) or a speech synthesis algorithm. The speech synthesis unit can generate speech in real time. For example, the speech synthesis unit takes the text received from the translation unit as input and generates speech using a TTS algorithm. The speech synthesis unit can also adjust the tone and speed of the speech to improve the naturalness of the speech. For example, the speech synthesis unit adjusts the tone and speed of the generated speech to provide a natural speech.

[0133] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires a voice indicating a user input for the result of the specific processing. The control unit 46A transmits the voice data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[0134] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating a voice, text data indicating a text, and image data indicating an image (e.g., still image data or video data) are input. The data generation model 58 infers the input inference data according to the instruction indicated by the prompt, and outputs the inference result in one or more data formats of voice data, text data, and image data. The data generation model 58 includes, for example, a text generation AI, an image generation AI, and a multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization. The specification processing unit 290 performs the above-mentioned specification processing while using the data generation model 58. The data generation model 58 may be a fine-tuned model so as to output an inference result from a prompt that does not include an instruction, and in this case, the data generation model 58 can output an inference result from a prompt that does not include an instruction. In the data processing device 12 and the like, the data generation model 58 includes a plurality of types, and the data generation model 58 includes an AI other than the generation AI. The AI ​​other than the generation AI is, for example, a linear regression, a logistic regression, a decision tree, a random forest, a support vector machine (SVM), a k-means clustering, a convolutional neural network (CNN), a recurrent neural network (RNN), a generative adversarial network (GAN), or a naive Bayes, and can perform various processes, but is not limited to such examples. The AI ​​may also be an AI agent. In addition, when the processes of each of the above-mentioned parts are performed by an AI, the processes are performed in part or in whole by the AI, but are not limited to such examples.In addition, processing performed by AI including the generation AI may be replaced with rule-based processing, and rule-based processing may be replaced with processing performed by AI including the generation AI.

[0135] Furthermore, the above-described processing by the data processing system 10 is executed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the smart device 14, but may also be executed by the specific processing unit 290 of the data processing device 12 and the control unit 46A of the smart device 14. Furthermore, the specific processing unit 290 of the data processing device 12 acquires or collects information required for processing from the smart device 14 or an external device, and the smart device 14 acquires or collects information required for processing from the data processing device 12 or an external device.

[0136] Each of the multiple elements including the above-mentioned voice recognition unit, translation unit, voice synthesis unit, minutes creation unit, ToDo management unit, and conference setup unit is realized, for example, in at least one of the smart device 14 and the data processing device 12. For example, the voice recognition unit captures the remarks of the participants using the microphone 38B of the smart device 14, and converts the voice into text by the control unit 46A. The translation unit translates the text into another language by the specific processing unit 290 of the data processing device 12. The voice synthesis unit outputs the translated content as voice using the speaker 40B of the smart device 14. The minutes creation unit records and saves the content of the conference by the specific processing unit 290 of the data processing device 12. The ToDo management unit manages tasks by the specific processing unit 290 of the data processing device 12 and updates the progress status. The conference setup unit sets the date and time of the next conference and participants by the specific processing unit 290 of the data processing device 12. The correspondence between each unit and the device or control unit is not limited to the above-mentioned example, and various changes are possible.

[0137] [Second embodiment] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0138] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0139] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN and / or a LAN.

[0140] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0141] The microphone 238 receives instructions and the like from the user by receiving voice uttered by the user. The microphone 238 captures the voice uttered by the user, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs the voice according to instructions from the processor 46.

[0142] Camera 42 is a small digital camera equipped with an optical system including a lens, an aperture, and a shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the user's surroundings (e.g., an imaging range defined by an angle of view equivalent to the width of the field of vision of an average healthy person).

[0143] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is performed in a secure state.

[0144] Fig. 4 shows an example of main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0145] The processor 28 reads the specific processing program 56 from the storage 32, and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0146] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290. The identification processing unit 290 can estimate the emotion of the user using the emotion identification model 59 and perform identification processing using the user's emotion. In the emotion estimation function (emotion identification function) using the emotion identification model 59, various estimations and predictions regarding the user's emotion, including estimation and prediction of the user's emotion, are performed, but are not limited to such examples. In addition, the estimation and prediction of the emotion also include, for example, analysis of the emotion.

[0147] In the smart glasses 214, the specific processing is performed by the processor 46. The storage 50 stores a specific processing program 60. The processor 46 reads the specific processing program 60 from the storage 50 and executes the read specific processing program 60 on the RAM 48. The specific processing is realized by the processor 46 operating as the control unit 46A in accordance with the specific processing program 60 executed on the RAM 48. Note that the smart glasses 214 have a data generation model and an emotion identification model similar to the data generation model 58 and the emotion identification model 59, and can also perform processing similar to that of the specific processing unit 290 using these models.

[0148] Note that a device other than the data processing device 12 may have the data generation model 58. For example, a server device may have the data generation model 58. In this case, the data processing device 12 obtains a processing result (such as a prediction result) using the data generation model 58 by communicating with the server device having the data generation model 58. In addition, the data processing device 12 may be a server device, or may be a terminal device (such as a mobile phone, a robot, or a home appliance) owned by a user.

[0149] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires a voice indicating a user input for the result of the specific processing. The control unit 46A transmits the voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[0150] The data generation model 58 is a so-called generative AI. An example of the data generation model 58 is a generative AI such as ChatGPT. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating a voice, text data indicating a text, and image data indicating an image (e.g., still image data or video data) is input. The data generation model 58 infers the input inference data according to the instruction indicated by the prompt, and outputs the inference result in one or more data formats of voice data, text data, and image data. The data generation model 58 includes, for example, a text generation AI, an image generation AI, and a multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization. The specification processing unit 290 performs the above-mentioned specification processing while using the data generation model 58. The data generation model 58 may be a fine-tuned model so as to output an inference result from a prompt that does not include an instruction, and in this case, the data generation model 58 can output an inference result from a prompt that does not include an instruction. In the data processing device 12 and the like, the data generation model 58 includes a plurality of types, and the data generation model 58 includes an AI other than the generation AI. The AI ​​other than the generation AI is, for example, a linear regression, a logistic regression, a decision tree, a random forest, a support vector machine (SVM), a k-means clustering, a convolutional neural network (CNN), a recurrent neural network (RNN), a generative adversarial network (GAN), or a naive Bayes, and can perform various processes, but is not limited to such examples. The AI ​​may be an AI agent. In addition, when the processes of each part described above are performed by an AI, the processes are performed in part or in whole by the AI, but are not limited to such examples. In addition, the processes performed by the AI ​​including the generation AI may be replaced with rule-based processes, and the rule-based processes may be replaced with processes performed by the AI ​​including the generation AI.

[0151] The data processing system 210 according to the second embodiment performs the same processing as the data processing system 10 according to the first embodiment. The processing by the data processing system 210 is executed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the smart glasses 214, but may be executed by the specific processing unit 290 of the data processing device 12 and the control unit 46A of the smart glasses 214. In addition, the specific processing unit 290 of the data processing device 12 acquires or collects information required for processing from the smart glasses 214 or an external device, and the smart glasses 214 acquires or collects information required for processing from the data processing device 12 or an external device.

[0152] Each of the multiple elements including the above-mentioned voice recognition unit, translation unit, voice synthesis unit, minutes creation unit, ToDo management unit, and conference setup unit is realized, for example, in at least one of the smart glasses 214 and the data processing device 12. For example, the voice recognition unit captures the remarks of the participants using the microphone 238 of the smart glasses 214, and converts the voice into text by the control unit 46A. The translation unit translates the text into another language by the specific processing unit 290 of the data processing device 12. The voice synthesis unit outputs the translated content as voice using the speaker 240 of the smart glasses 214. The minutes creation unit records and saves the content of the conference by the specific processing unit 290 of the data processing device 12. The ToDo management unit manages tasks by the specific processing unit 290 of the data processing device 12 and updates the progress status. The conference setup unit sets the date and time of the next conference and participants by the specific processing unit 290 of the data processing device 12. The correspondence between each unit and the device or control unit is not limited to the above-mentioned example, and various changes are possible.

[0153] [Third embodiment] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0154] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0155] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN and / or a LAN.

[0156] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0157] The microphone 238 receives instructions and the like from the user by receiving voice uttered by the user. The microphone 238 captures the voice uttered by the user, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs the voice according to instructions from the processor 46.

[0158] Camera 42 is a small digital camera equipped with an optical system including a lens, an aperture, and a shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the user's surroundings (e.g., an imaging range defined by an angle of view equivalent to the width of the field of vision of an average healthy person).

[0159] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is performed in a secure state.

[0160] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0161] The processor 28 reads the specific processing program 56 from the storage 32, and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0162] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290. The identification processing unit 290 can estimate the emotion of the user using the emotion identification model 59 and perform identification processing using the user's emotion. In the emotion estimation function (emotion identification function) using the emotion identification model 59, various estimations and predictions regarding the user's emotion, including estimation and prediction of the user's emotion, are performed, but are not limited to such examples. In addition, the estimation and prediction of the emotion also include, for example, analysis of the emotion.

[0163] In the headset type terminal 314, the specific processing is performed by the processor 46. The storage 50 stores a specific program 60. The processor 46 reads the specific program 60 from the storage 50, and executes the read specific program 60 on the RAM 48. The specific processing is realized by the processor 46 operating as the control unit 46A in accordance with the specific program 60 executed on the RAM 48. Note that the headset type terminal 314 has a data generation model and an emotion identification model similar to the data generation model 58 and the emotion identification model 59, and can also perform processing similar to that of the identification processing unit 290 using these models.

[0164] Note that a device other than the data processing device 12 may have the data generation model 58. For example, a server device may have the data generation model 58. In this case, the data processing device 12 obtains a processing result (such as a prediction result) using the data generation model 58 by communicating with the server device having the data generation model 58. In addition, the data processing device 12 may be a server device, or may be a terminal device (such as a mobile phone, a robot, or a home appliance) owned by a user.

[0165] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input for the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[0166] The data generation model 58 is a so-called generative AI. An example of the data generation model 58 is a generative AI such as ChatGPT. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating a voice, text data indicating a text, and image data indicating an image (e.g., still image data or video data) is input. The data generation model 58 infers the input inference data according to the instruction indicated by the prompt, and outputs the inference result in one or more data formats of voice data, text data, and image data. The data generation model 58 includes, for example, a text generation AI, an image generation AI, and a multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization. The specification processing unit 290 performs the above-mentioned specification processing while using the data generation model 58. The data generation model 58 may be a fine-tuned model so as to output an inference result from a prompt that does not include an instruction, and in this case, the data generation model 58 can output an inference result from a prompt that does not include an instruction. In the data processing device 12 and the like, the data generation model 58 includes a plurality of types, and the data generation model 58 includes an AI other than the generation AI. The AI ​​other than the generation AI is, for example, a linear regression, a logistic regression, a decision tree, a random forest, a support vector machine (SVM), a k-means clustering, a convolutional neural network (CNN), a recurrent neural network (RNN), a generative adversarial network (GAN), or a naive Bayes, and can perform various processes, but is not limited to such examples. The AI ​​may be an AI agent. In addition, when the processes of each part described above are performed by an AI, the processes are performed in part or in whole by the AI, but are not limited to such examples. In addition, the processes performed by the AI ​​including the generation AI may be replaced with rule-based processes, and the rule-based processes may be replaced with processes performed by the AI ​​including the generation AI.

[0167] The data processing system 310 according to the third embodiment performs the same processing as the data processing system 10 according to the first embodiment. The processing by the data processing system 310 is executed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the headset type terminal 314, but may be executed by the specific processing unit 290 of the data processing device 12 and the control unit 46A of the headset type terminal 314. In addition, the specific processing unit 290 of the data processing device 12 acquires or collects information required for processing from the headset type terminal 314 or an external device, and the headset type terminal 314 acquires or collects information required for processing from the data processing device 12 or an external device.

[0168] Each of the multiple elements including the above-mentioned voice recognition unit, translation unit, voice synthesis unit, minutes creation unit, ToDo management unit, and conference setup unit is realized, for example, in at least one of the headset type terminal 314 and the data processing device 12. For example, the voice recognition unit captures the speech of the participants using the microphone 238 of the headset type terminal 314, and converts the speech into text by the control unit 46A. The translation unit translates the text into another language by the specific processing unit 290 of the data processing device 12. The voice synthesis unit outputs the translated content as voice using the speaker 240 of the headset type terminal 314. The minutes creation unit records and saves the content of the conference by the specific processing unit 290 of the data processing device 12. The ToDo management unit manages tasks by the specific processing unit 290 of the data processing device 12 and updates the progress status. The conference setup unit sets the date and time of the next conference and participants by the specific processing unit 290 of the data processing device 12. The correspondence between each unit and the device or control unit is not limited to the above-mentioned example, and various changes are possible.

[0169] [Fourth embodiment] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[0170] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0171] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN and / or a LAN.

[0172] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[0173] The microphone 238 receives instructions and the like from the user by receiving voice uttered by the user. The microphone 238 captures the voice uttered by the user, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs the voice according to instructions from the processor 46.

[0174] Camera 42 is a small digital camera equipped with an optical system including a lens, an aperture, and a shutter, and an imaging element such as a CMOS image sensor or a CCD image sensor, and captures images of the user's surroundings (e.g., an imaging range defined by an angle of view equivalent to the width of the field of vision of a typical healthy person).

[0175] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is performed in a secure state.

[0176] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and legs. The posture and behavior of the robot 414 are controlled by controlling the motors of the arms, hands, and legs. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[0177] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0178] The processor 28 reads the specific processing program 56 from the storage 32, and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0179] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290. The identification processing unit 290 can estimate the emotion of the user using the emotion identification model 59 and perform identification processing using the user's emotion. In the emotion estimation function (emotion identification function) using the emotion identification model 59, various estimations and predictions regarding the user's emotion, including estimation and prediction of the user's emotion, are performed, but are not limited to such examples. In addition, the estimation and prediction of the emotion also include, for example, analysis of the emotion.

[0180] In the robot 414, the specific processing is performed by the processor 46. The storage 50 stores a specific program 60. The processor 46 reads out the specific program 60 from the storage 50, and executes the specific program 60 on the RAM 48. The specific processing is realized by the processor 46 operating as the control unit 46A in accordance with the specific program 60 executed on the RAM 48. The robot 414 has a data generation model and an emotion identification model similar to the data generation model 58 and the emotion identification model 59, and can also perform processing similar to that of the specific processing unit 290 using these models.

[0181] Note that a device other than the data processing device 12 may have the data generation model 58. For example, a server device may have the data generation model 58. In this case, the data processing device 12 obtains a processing result (such as a prediction result) using the data generation model 58 by communicating with the server device having the data generation model 58. In addition, the data processing device 12 may be a server device, or may be a terminal device (such as a mobile phone, a robot, or a home appliance) owned by a user.

[0182] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires a voice indicating a user input for the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[0183] The data generation model 58 is a so-called generative AI. An example of the data generation model 58 is a generative AI such as ChatGPT. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating a voice, text data indicating a text, and image data indicating an image (e.g., still image data or video data) is input. The data generation model 58 infers the input inference data according to the instruction indicated by the prompt, and outputs the inference result in one or more data formats of voice data, text data, and image data. The data generation model 58 includes, for example, a text generation AI, an image generation AI, and a multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization. The specification processing unit 290 performs the above-mentioned specification processing while using the data generation model 58. The data generation model 58 may be a fine-tuned model so as to output an inference result from a prompt that does not include an instruction, and in this case, the data generation model 58 can output an inference result from a prompt that does not include an instruction. In the data processing device 12 and the like, the data generation model 58 includes a plurality of types, and the data generation model 58 includes an AI other than the generation AI. The AI ​​other than the generation AI is, for example, a linear regression, a logistic regression, a decision tree, a random forest, a support vector machine (SVM), a k-means clustering, a convolutional neural network (CNN), a recurrent neural network (RNN), a generative adversarial network (GAN), or a naive Bayes, and can perform various processes, but is not limited to such examples. The AI ​​may be an AI agent. In addition, when the processes of each part described above are performed by an AI, the processes are performed in part or in whole by the AI, but are not limited to such examples. In addition, the processes performed by the AI ​​including the generation AI may be replaced with rule-based processes, and the rule-based processes may be replaced with processes performed by the AI ​​including the generation AI.

[0184] The data processing system 410 according to the fourth embodiment performs the same processing as the data processing system 10 according to the first embodiment. The processing by the data processing system 410 is executed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the robot 414, but may be executed by the specific processing unit 290 of the data processing device 12 and the control unit 46A of the robot 414. In addition, the specific processing unit 290 of the data processing device 12 acquires or collects information required for processing from the robot 414 or an external device, etc., and the robot 414 acquires or collects information required for processing from the data processing device 12 or an external device, etc.

[0185] Each of the multiple elements including the above-mentioned voice recognition unit, translation unit, voice synthesis unit, minutes creation unit, ToDo management unit, and conference setup unit is realized, for example, by at least one of the robot 414 and the data processing device 12. For example, the voice recognition unit captures the speech of the participants using the microphone 238 of the robot 414, and converts the speech into text by the control unit 46A. The translation unit translates the text into another language by the specific processing unit 290 of the data processing device 12. The voice synthesis unit outputs the translated content as voice using the speaker 240 of the robot 414. The minutes creation unit records and saves the content of the conference by the specific processing unit 290 of the data processing device 12. The ToDo management unit manages tasks by the specific processing unit 290 of the data processing device 12 and updates the progress status. The conference setup unit sets the date and time of the next conference and participants by the specific processing unit 290 of the data processing device 12. The correspondence between each unit and the device or control unit is not limited to the above-mentioned example, and various changes are possible.

[0186] The emotion identification model 59 as an emotion engine may determine the emotion of the user according to a specific mapping. Specifically, the emotion identification model 59 may determine the emotion of the user according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the emotion of the robot, and the identification processing unit 290 may perform identification processing using the emotion of the robot.

[0187] FIG. 9 is a diagram showing an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive emotions are arranged. The more outside the concentric circles, the more emotions that represent states and actions that arise from a state of mind are arranged. Emotions are a concept that includes emotions and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions that occur in the brain are arranged. On the right side of the concentric circles, emotions that are generally induced by situational judgment are arranged. On the upper and lower sides of the concentric circles, emotions that are generally generated from reactions that occur in the brain and are induced by situational judgment are arranged. In addition, on the upper side of the concentric circles, emotions of "pleasure" are arranged, and on the lower side, emotions of "discomfort" are arranged. In this way, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[0188] These emotions are distributed in the 3 o'clock direction of emotion map 400 and usually fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[0189] The inside of emotion map 400 represents what is going on inside one's mind, and the outside of emotion map 400 represents behavior, so the further out on emotion map 400 you go, the more visible the emotions become (the more they are expressed in behavior).

[0190] Here, human emotions are based on various balances such as posture and blood sugar level, and when these balances are far from the ideal, it indicates an unpleasant state, and when they are close to the ideal, it indicates a pleasant state. Emotions can also be created for robots, cars, motorcycles, etc., based on various balances such as posture and battery level, so that when these balances are far from the ideal, it indicates an unpleasant state, and when they are close to the ideal, it indicates a pleasant state. The emotion map may be generated, for example, based on the emotion map of Dr. Mitsuyoshi (Research on speech emotion recognition and emotion brain physiological signal analysis system, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). On the left half of the emotion map, emotions belonging to an area called "reaction" where sensation is dominant are lined up. On the right half of the emotion map, emotions belonging to an area called "situation" where situation recognition is dominant are lined up.

[0191] The emotion map defines two emotions that promote learning. The first is the negative emotion around the middle of "repentance" or "remorse" on the situation side. In other words, this is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the positive emotion around "desire" on the response side. In other words, this is when the robot has positive feelings such as "I want more" or "I want to know more."

[0192] The emotion identification model 59 inputs the user input to a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the emotion of the user. This neural network is pre-trained based on multiple learning data that are combinations of the user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in Fig. 10. Fig. 10 shows an example in which multiple emotions, "relief," "calm," and "encouraging," have similar emotion values.

[0193] In the above embodiment, an example was given in which a specific process is performed by one computer 22, but the technology disclosed herein is not limited to this, and distributed processing of the specific process may be performed by multiple computers, including computer 22.

[0194] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable computer-readable non-transitory storage medium such as a Universal Serial Bus (USB) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[0195] In addition, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 upon request from the data processing device 12.

[0196] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[0197] As the hardware resource for executing the specific process, various processors as shown below can be used. An example of the processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing the specific process by executing software, i.e., a program. Another example of the processor is a dedicated electric circuit, which is a processor having a circuit configuration designed exclusively for executing a specific process, such as a Field-Programmable Gate Array (FPGA), a Programmable Logic Device (PLD), or an Application Specific Integrated Circuit (ASIC). Each processor has a built-in or connected memory, and each processor executes the specific process by using the memory.

[0198] The hardware resource that executes the specific process may be one of these various processors, or may be a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific process may be a single processor.

[0199] As an example of a configuration using one processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a configuration using a processor that realizes the functions of the entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[0200] Furthermore, more specifically, the hardware structure of these various processors can be an electric circuit that combines circuit elements such as semiconductor elements. The specific processes described above are merely examples. It goes without saying that unnecessary steps may be deleted, new steps may be added, or the order of processes may be changed without departing from the spirit of the invention.

[0201] In the above example, the first to fourth embodiments are described separately, but some or all of these embodiments may be combined. The smart device 14, the smart glasses 214, the headset terminal 314, and the robot 414 are only examples, and they may be combined with each other, or may be other devices. In the above example, the first and second embodiments are described separately, but they may be combined.

[0202] The above description and illustrations are detailed descriptions of the parts related to the technology of the present disclosure, and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, function, action, and effect is an example of the configuration, function, action, and effect of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replaced with respect to the description and illustrations shown above, within the scope of the gist of the technology of the present disclosure. In addition, in order to avoid confusion and to facilitate understanding of the parts related to the technology of the present disclosure, the description and illustrations shown above omit explanations of technical common sense that do not require particular explanation in enabling the implementation of the technology of the present disclosure.

[0203] All publications, patent applications, and standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, and standard was specifically and individually indicated to be incorporated by reference.

[0204] (Appendix 1) a speech recognition unit that recognizes speech of participants in a conference; a translation unit that translates the utterance recognized by the speech recognition unit; A voice synthesis unit that outputs the content translated by the translation unit as voice. A system characterized by: (Appendix 2) Have a minutes-taking department to take minutes of meetings 2. The system of claim 1, (Appendix 3) Equipped with a ToDo management section for managing ToDos 2. The system of claim 1, (Appendix 4) Equipped with a conference setup section to set up the next conference 2. The system of claim 1, (Appendix 5) The voice recognition unit Transcribe what meeting participants say 2. The system of claim 1, (Appendix 6) The translation unit is Translate transcribed speech into other languages ​​in real time 2. The system of claim 1, (Appendix 7) The voice synthesis unit Output the translated content as audio 2. The system of claim 1, (Appendix 8) The voice recognition unit Estimate the emotion of the participant and adjust the accuracy of the speech recognition based on the estimated emotion 2. The system of claim 1, (Appendix 9) The voice recognition unit Eliminate background noise when participants are speaking to make it easier to understand what is being said 2. The system of claim 1, (Appendix 10) The voice recognition unit Dynamically adjust speech recognition parameters according to participants' speaking rates 2. The system of claim 1, (Appendix 11) The voice recognition unit Estimate the emotion of the participant and prioritize speech recognition based on the estimated emotion 2. The system of claim 1, (Appendix 12) The voice recognition unit Summarize what participants say in real time and highlight key points 2. The system of claim 1, (Appendix 13) The voice recognition unit Transcribing participants' comments in multiple languages ​​simultaneously and providing them to participants of different languages 2. The system of claim 1, (Appendix 14) The translation unit is Estimate the participant's emotions and adjust the translation based on the estimated emotions 2. The system of claim 1, (Appendix 15) The translation unit is Dynamically update dictionaries to properly translate technical and industry-specific terminology as you translate 2. The system of claim 1, (Appendix 16) The translation unit is When translating, an algorithm is applied that takes context into account to select the appropriate translation 2. The system of claim 1, (Appendix 17) The translation unit is Estimate participants' emotions and prioritize translations based on the estimated emotions 2. The system of claim 1, (Appendix 18) The translation unit is Output translated content in multiple languages ​​simultaneously to serve multilingual participants 2. The system of claim 1, (Appendix 19) The translation unit is Summarize the translated content and highlight the important points 2. The system of claim 1, (Appendix 20) The voice synthesis unit Estimate the emotion of the participant and adjust the tone and rate of speech based on the estimated emotion 2. The system of claim 1, (Appendix 21) The voice synthesis unit Dynamically adjust parameters to reproduce the characteristics of a speaker's voice during speech synthesis 2. The system of claim 1, (Appendix 22) The voice synthesis unit When synthesizing speech, the system adjusts background sounds appropriately to produce speech that is easy to listen to. 2. The system of claim 1, (Appendix 23) The voice synthesis unit Estimate participants' emotions and prioritize audio based on the estimated emotions 2. The system of claim 1, (Appendix 24) The voice synthesis unit Outputting synthesized speech in multiple languages ​​simultaneously for multilingual participants 2. The system of claim 1, (Appendix 25) The voice synthesis unit Summarize synthesized content and highlight key points 2. The system of claim 1, (Appendix 26) The minutes preparation department comprises: Infer participants' emotions and adjust the presentation of the minutes based on the inferred emotions 2. The system of claim 1, (Appendix 27) The minutes preparation department comprises: When taking minutes, adjust the level of detail based on the importance of what was said 2. The system of claim 1, (Appendix 28) The minutes preparation department comprises: Generate appropriate summaries when taking minutes by considering the context of what is said 2. The system of claim 1, (Appendix 29) The minutes preparation department comprises: Estimate participants' emotions and prioritize minutes based on the estimated emotions 2. The system of claim 1, (Appendix 30) The minutes preparation department comprises: Create meeting minutes in multiple languages ​​simultaneously and provide them to participants with different languages 2. The system of claim 1, (Appendix 31) The minutes preparation department comprises: Summarize the minutes and highlight key points 2. The system of claim 1, (Appendix 32) The ToDo management unit includes: Estimate participants' emotions and adjust the priorities of the to-do list based on the estimated emotions 2. The system of claim 1, (Appendix 33) The ToDo management unit includes: When managing your to-dos, adjust the list detail based on the importance of the tasks 2. The system of claim 1, (Appendix 34) The ToDo management unit includes: Update task progress in real time when managing ToDos 2. The system of claim 1, (Appendix 35) The ToDo management unit includes: Estimate participants' emotions and adjust how the to-do list is displayed based on the estimated emotions 2. The system of claim 1, (Appendix 36) The ToDo management unit includes: Create to-do lists in multiple languages ​​simultaneously and provide them to participants of different languages 2. The system of claim 1, (Appendix 37) The ToDo management unit includes: Summarize your to-do list and highlight key points 2. The system of claim 1, (Appendix 38) The conference setup unit: Estimate participants' emotions and schedule the next meeting based on the estimated emotions 2. The system of claim 1, (Appendix 39) The conference setup unit: When setting up a meeting, choose an appropriate date and time based on the participants' schedules. 2. The system of claim 1, (Appendix 40) The conference setup unit: When setting up a conference, select the appropriate participants by referring to the past participation history of the participants 2. The system of claim 1, (Appendix 41) The conference setup unit: Estimate participants' emotions and prioritize meetings based on the estimated emotions 2. The system of claim 1, (Appendix 42) The conference setup unit: Simultaneous meeting setup in multiple languages ​​for participants of different languages 2. The system of claim 1, (Appendix 43) The conference setup unit: Create a summary of the meeting setup and highlight key points 2. The system of claim 1, [Explanation of symbols]

[0205] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot

Claims

1. a speech recognition unit that recognizes speech of participants in a conference; a translation unit that translates the utterance recognized by the speech recognition unit; A voice synthesis unit that outputs the content translated by the translation unit as voice. A system characterized by:

2. Have a minutes-taking department to take minutes of meetings 2. The system of claim 1 .

3. Equipped with a ToDo management section for managing ToDos 2. The system of claim 1 .

4. Equipped with a conference setup section to set up the next conference 2. The system of claim 1 .

5. The voice recognition unit Transcribe what meeting participants say 2. The system of claim 1 .

6. The translation unit is Translate transcribed speech into other languages ​​in real time 2. The system of claim 1 .

7. The voice synthesis unit Output the translated content as audio 2. The system of claim 1 .

8. The voice recognition unit Estimate the emotion of the participant and adjust the accuracy of the speech recognition based on the estimated emotion 2. The system of claim 1 .

9. The voice recognition unit Eliminate background noise when participants are speaking to make it easier to understand what is being said 2. The system of claim 1 .

10. The voice recognition unit Dynamically adjust speech recognition parameters according to participants' speaking rates 2. The system of claim 1 .

Images