system

Abstract

We provide the system. [Solution] In a conference system, Means of collecting audio data, A means of transmitting the collected audio data to the server in real time, A speech recognition means for converting transmitted audio data into text data, A translation method for translating converted text data into a different language, A filtering means for detecting and masking or replacing confidential information from translated text data, A means of sending the final text data to the receiving terminal and displaying it in real time, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a persona chatbot control method performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance as a response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Currently, the real - time captioning function in a conference system has problems such as low speech recognition accuracy, misunderstandings due to misrecognition, inaccurate translation, and even leakage of confidential information outside the company. Therefore, it is difficult for people with hearing impairments and participants in multilingual conferences to obtain accurate information in real time. The object of the present invention is to solve such problems and provide a conference system that simultaneously improves speech recognition accuracy, translation accuracy, and protects confidential information.

Means for Solving the Problems

[0005] The present invention relates to a conference system comprising means for collecting audio data, means for transmitting the collected audio data to a server in real time, speech recognition means for converting the transmitted audio data into text data, translation means for translating the converted text data into different languages, filtering means for detecting, masking, or replacing confidential information from the translated text data, and means for transmitting the final text data to a receiving terminal and displaying it in real time. Furthermore, the system includes means for dividing the audio data into appropriate chunks, and the speech recognition means uses an artificial intelligence model. This improves the accuracy and speed of speech recognition, enables accurate translation and protection of confidential information, and provides a system in which the hearing impaired and participants in multilingual conferences can obtain accurate information in real time.

[0006] "Audio data" refers to the digital representation of audio signals collected in a conference system.

[0007] "Speech recognition" is a technology that analyzes collected speech data and converts it into corresponding text data.

[0008] Translation is the process of converting text data expressed in one language into a different language.

[0009] "Confidential information" refers to important information shared during a meeting that is restricted from being made public.

[0010] "Filtering" is the process of detecting sensitive information and masking or replacing it.

[0011] A "server" is a central computing device that processes audio data, performs speech recognition, and translates and filters text data.

[0012] A "terminal" is a device connected to a conference system that collects and transmits audio data, and receives and displays the final text data.

[0013] A "chunk" is an appropriate size unit into which audio data is divided.

[0014] An "artificial intelligence model" is an algorithm that learns from large amounts of data and automatically performs specific tasks such as speech recognition, translation, and filtering. [Brief explanation of the drawing]

[0015] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] This is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] This is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13]It is a sequence diagram showing the processing flow of the data processing system in Embodiment 2 when the emotion engine is combined. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when the emotion engine is combined.

Mode for Carrying Out the Invention

[0016] Hereinafter, an example of an embodiment of the system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0017] First, the language used in the following description will be explained.

[0018] In the following embodiments, the numbered processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0019] In the following embodiments, the numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0020] In the following embodiments, the numbered storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, and the like.

[0021] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0022] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0023] [First Embodiment]

[0024] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0025] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0026] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0027] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0028] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0029] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0030] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0031] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0032] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0033] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0034] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0035] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0036] This invention relates to a system that combines speech recognition, translation, confidential information protection, and real-time captioning in a conference system. The system aims to collect audio data, transmit it to a server, perform speech recognition to convert it into text data, translate and filter it as needed, and finally display it on a terminal in real time.

[0037] System Overview

[0038] This system is configured as follows: First, when a user speaks during a meeting, their voice is captured in real time by the terminal. The captured voice data is appropriately segmented and sent to the server. The server receives the voice data and converts it into text data using an AI speech recognition model. The converted text data is translated into different languages ​​by translation means as needed. Subsequently, sensitive information is detected and masked or replaced in the text data by filtering means. The final text data is sent back to the terminal and displayed to the user in real time.

[0039] Specific example

[0040] Example 1: Japanese meeting

[0041] Let's say User A says "The next project starts next month" during a Zoom meeting in Japanese. This audio is captured by the device, split into 1-second segments, and sent to the server. The server inputs this audio into an AI speech recognition model and converts it into text data: "The next project starts next month." This text data is checked by a filtering mechanism, but since it does not contain any particularly confidential information, it is sent back to the device as is. The device displays this text data as real-time subtitles on the user's screen.

[0042] Example 2: A meeting requiring translation from Japanese to English

[0043] In another scenario, consider a situation where user B says in Japanese, "We have achieved our sales targets." This audio is also captured by the terminal, split, and sent to the server. The server inputs the audio into an AI speech recognition model and generates text data that reads, "We have achieved our sales targets." This text data is then translated into English by a translation tool, resulting in "We have achieved our sales targets." This translation is also checked by a filtering tool and determined not to contain any confidential information. The final text data is then sent to the terminal and displayed as English subtitles.

[0044] Program processing

[0045] 1. Audio data collection: The device captures the user's spoken audio in real time, divides it into appropriate-sized chunks, and sends them to the server.

[0046] 2. Speech Recognition: The server receives the speech data and inputs it into the AI ​​speech recognition model to convert it into text data.

[0047] 3. Translation: If necessary, the converted text data is input into a generated AI translation model and translated into the specified language.

[0048] 4. Filtering: The server detects and masks or replaces confidential company information within the text data.

[0049] 5. Subtitle display: The final text data is sent to the device and displayed on the user's screen in real time.

[0050] In this way, the present invention realizes a system that accurately recognizes speech during meetings, translates it as needed, and displays subtitles in real time while protecting confidential information.

[0051] The following describes the processing flow.

[0052] Step 1:

[0053] Collection of audio data

[0054] User: Speaks during the meeting.

[0055] Terminal: To capture user speech in real time, the system uses the conferencing system's API to obtain the audio stream. The acquired audio data is then divided into segments of a fixed length (e.g., every second).

[0056] Terminal: Stores the divided audio data chunks in a buffer and prepares them for transmission to the server.

[0057] Step 2:

[0058] Sending audio data

[0059] Terminal: Sends audio data chunks stored in the buffer to the server at the appropriate time. Considering network conditions, it is mindful of minimizing latency.

[0060] Step 3:

[0061] Speech recognition

[0062] Server: Receives audio data chunks sent from the terminal.

[0063] Server: Inputs the received audio data chunks into an AI speech recognition model (e.g., Google® Speech-to-Text API) to generate the corresponding text data.

[0064] Server: The server processes the text data obtained from the speech recognition model, performing post-processing such as noise reduction and string normalization to improve accuracy.

[0065] Step 4:

[0066] Translation (if necessary)

[0067] Server: Compares the language in which the text data was generated with the display language specified by the user to determine whether translation is necessary.

[0068] Server: When translation is needed, it inputs text data into an AI translation model (e.g., Google Translate API) and translates it into the specified language.

[0069] Server: Review the translation results and make any necessary adjustments.

[0070] Step 5:

[0071] Filtering of confidential company information

[0072] Server: Uses filtering mechanisms to detect confidential company information within the generated text data (including after translation).

[0073] Server: Mask or replace detected confidential information with appropriate tags. This prevents the leakage of sensitive information.

[0074] Server: Generates the final filtered text data.

[0075] Step 6:

[0076] Subtitle data transmission and display

[0077] Server: Prepares to send the final text data to the terminal.

[0078] Server: Sends the final text data to the terminal.

[0079] Terminal: Receives text data sent from the server.

[0080] Terminal: Displays received text data on the user's screen in real time. This allows the user to see what is being said during the meeting as subtitles.

[0081] This processing flow enables the system to achieve highly accurate speech recognition and translation, providing users with real-time subtitles while appropriately protecting confidential information.

[0082] (Example 1)

[0083] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0084] In conferencing systems, there is a demand for real-time subtitle display while simultaneously supporting multiple languages ​​and protecting confidential information. However, current technology does not integrate the processes of speech recognition, translation, and filtering, making real-time processing difficult. Furthermore, if the efficient splitting and transmission of audio data are not performed properly, processing delays and reduced accuracy can occur.

[0085] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0086] In this invention, the server includes means for transmitting audio data to the server in real time, speech recognition means for converting the transmitted audio data into text data, translation means for translating the converted text data into different languages, filtering means for detecting, masking, or replacing confidential information from the translated text data, means for transmitting the final text data to a receiving terminal and displaying it in real time, and means for dividing the audio data into appropriate sizes. This enables real-time subtitle display in a conference system while achieving multilingual support and protection of confidential information.

[0087] "Audio data" refers to audio signals or digital data collected during meetings or conversations.

[0088] A "server" is a data processing device or computer system on a network that receives and processes audio data.

[0089] "Text data" refers to character information converted from speech data by speech recognition technology.

[0090] "Speech recognition means" refers to recognition technology or algorithms for converting speech data into text data.

[0091] "Translation methods" refer to techniques or algorithms that convert text data into different languages.

[0092] "Filtering means" refers to techniques or algorithms that detect and mask or replace confidential information from text data.

[0093] A "receiving terminal" is a device that receives the final text data sent from the server and displays it to the user.

[0094] "Means for displaying in real time" refers to a technology or device that displays text data that has undergone conversion and filtering processes without delay.

[0095] "Means for dividing into appropriate sizes" refers to techniques or devices for dividing audio data into appropriate time intervals so that it can be easily processed.

[0096] Modes for carrying out the invention

[0097] This invention relates to a conferencing system that combines speech recognition, translation, confidential information protection, and real-time captioning. This system uses specific hardware and software to collect, transmit, analyze, convert, filter, and display audio data.

[0098] Collection and transmission of audio data

[0099] When a user speaks during a meeting, the device captures their audio in real time. For example, Zoom or Microsoft Teams® can be used as meeting software. The captured audio data is divided into appropriate sizes (e.g., every second) and sent from the device to the server. It is desirable that the audio data be compressed during transmission.

[0100] Speech recognition

[0101] The server receives audio data sent from the terminal in real time. The audio data is input into an AI speech recognition model (e.g., Google Speech-to-Text API) and converted into text data. For example, the statement "The next project starts next month" is converted into the text data "The next project starts next month".

[0102] translation

[0103] The converted text data is then fed into translation models (e.g., DeepL, Google Translate) as needed, and translated into different languages. For example, the Japanese phrase "We have achieved our sales targets" is translated into English as "We have achieved our sales targets."

[0104] filtering

[0105] The server inspects the translated text data and detects sensitive information. Using natural language processing tools (e.g., spaCy, Microsoft Azure® Text Analytics), it identifies specific sensitive information and masks or replaces the relevant sections. For example, "We achieved our sales target" might be masked to "Our sales target is..."

[0106] Subtitle display

[0107] The final text data is sent from the server to the terminal. The terminal displays the received text data as real-time subtitles on the user's screen. Subtitle display software such as OBS Studio is used for display.

[0108] Specific example

[0109] Example 1: Japanese meeting

[0110] User A: In a Japanese Zoom meeting, says, "The next project will start next month."

[0111] Terminal: Captures audio in real time, splits it into 1-second segments, and sends them to the server.

[0112] Server: Receives the audio and uses the Google Speech-to-Text API to convert it into text data that says, "The next project starts next month."

[0113] Server: Inspects the text data and, since it does not contain any particularly sensitive information, sends it to the terminal as is.

[0114] Terminal: Displays text data as real-time subtitles.

[0115] Example 2: A meeting requiring translation from Japanese to English

[0116] User B: Says in Japanese, "We have achieved our sales target."

[0117] Terminal: Captures audio and sends it to the server.

[0118] Server: Inputs the audio into an AI speech recognition model and generates text data that says, "Sales target achieved."

[0119] Server: Inputs text data into DeepL and obtains the English translation "We have achieved our sales targets".

[0120] Server: The translation results are checked and determined not to contain any confidential information.

[0121] Device: Display as English subtitles.

[0122] Example of a prompt

[0123] Speech Recognition and Real-Time Display for Japanese Meetings: "Please describe a program that converts Japanese audio during a meeting into text data and displays it in real time."

[0124] Translation and display from Japanese to English: "Please explain the process of translating Japanese audio into English and displaying it as subtitles in real time."

[0125] In this way, the present invention provides a means for efficiently and effectively conducting meetings by integrating speech recognition, translation, confidential information protection, and real-time captioning into a single system within a conference system.

[0126] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0127] Step 1:

[0128] Collection and capture of audio data

[0129] Input: Audio of a user speaking during a meeting.

[0130] Specific action: The user speaks through conferencing software such as Zoom or Teams.

[0131] Output: Audio data captured by the device.

[0132] Processing details: The device captures the user's speech in real time using the microphone. This audio data is saved in digital format and immediately split.

[0133] Step 2:

[0134] Splitting and transmitting audio data

[0135] Input: Captured audio data.

[0136] Specific operation: The device divides the captured audio data into 1-second segments. It also compresses the audio data to reduce unnecessary data.

[0137] Output: Split and compressed audio data.

[0138] Processing details: The terminal divides the captured audio data into appropriate chunks (every second), compresses the data, and sends it to the server. This process improves communication efficiency.

[0139] Step 3:

[0140] Receiving and recognizing audio data

[0141] Input: Split audio data sent from the device.

[0142] Specific operation: The server receives this data and passes it to an AI speech recognition model (e.g., Google Speech-to-Text API).

[0143] Output: Text data.

[0144] Processing details: The server inputs the received audio data into an AI speech recognition model and converts it into text data in real time. For example, "The next project starts next month" is generated as text data.

[0145] Step 4:

[0146] Text data translation

[0147] Input: Text data generated by speech recognition.

[0148] Specific operation: The server inputs text data into a translation model (e.g., Google Translate, DeepL) according to the requirements.

[0149] Output: Translated text data.

[0150] Processing details: The converted text data is input into a translation model and translated into the specified language. For example, "We have achieved our sales targets" is translated as "We have achieved our sales targets."

[0151] Step 5:

[0152] Filtering of translated text data

[0153] Input: Translated text data.

[0154] Specific operation: The server detects sensitive information within the text data using filtering tools (e.g., spaCy, Microsoft Azure Text Analytics).

[0155] Output: Text data with confidential information masked or replaced.

[0156] Processing details: Inspect the translated text data and mask any confidential information found (e.g., "Sales targets are") or replace it (e.g., "Top Secret Data" → "Important Data").

[0157] Step 6:

[0158] Sending the final text data to the receiving terminal

[0159] Input: Filtered text data.

[0160] Specific operation: The server sends the final text data to the receiving terminal.

[0161] Output: Text data sent to the receiving terminal.

[0162] Processing details: The final text data is sent from the server to the receiving terminal. The sent data is processed at the receiving terminal.

[0163] Step 7:

[0164] Real-time subtitle display

[0165] Input: The final text data sent from the server.

[0166] Specific operation: The receiving terminal displays this text data on its screen in real time. For example, subtitles can be displayed using OBS Studio or similar software.

[0167] Output: Real-time subtitles displayed on the user's screen.

[0168] Processing details: The receiving terminal displays the final text data as real-time subtitles. This makes it easier for users to understand the meeting content.

[0169] (Application Example 1)

[0170] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0171] Communication among passengers in autonomous vehicles presents challenges such as language barriers and the risk of confidential information leakage. Furthermore, while real-time speech recognition, translation, and the protection and display of confidential information are necessary, there is a lack of systems that can efficiently achieve these goals. This challenge is particularly pronounced in multilingual environments and business meetings where the risk of information leakage is high.

[0172] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0173] In this invention, the server includes means for collecting voice data, means for transmitting the collected voice data to the server in real time, and voice recognition means that uses a generative AI model to convert the voice data into text data. This enables support for communication between multiple languages ​​and protection of confidential information.

[0174] A "conference system and in-vehicle communication system" is a system that has functions such as voice data collection, speech recognition, translation, confidential information protection, and real-time display, and operates effectively in multilingual environments and situations with a high risk of information leakage.

[0175] "Means for collecting audio data" refers to devices such as microphones and digital audio recorders used to capture audio in real time from inside a vehicle or conference room.

[0176] "Means of sending to the server in real time" refers to network interfaces and data transfer protocols for sending captured audio data to the server without delay.

[0177] "Speech recognition means" refers to software or hardware used to convert speech data into text data, particularly those that utilize generative AI models.

[0178] "Translation means" refers to software or generative AI models used to translate text data obtained by speech recognition means into different languages.

[0179] "Filtering means" refers to algorithms or software used to detect confidential information from translated text data and to mask or replace that information.

[0180] "Means for sending to a receiving terminal and displaying in real time" refers to programs or hardware that send the final text data to a client device and display it on that device in real time.

[0181] "Means for displaying text data in real time on an in-vehicle display" refers to a device that has an interface function for displaying text data in real time on a screen or monitor installed inside a vehicle.

[0182] A "generative AI model" is an artificial intelligence algorithm trained to perform tasks such as speech recognition, translation, and sensitive information detection.

[0183] This invention is a system for supporting communication within autonomous vehicles and conference systems. The "Conference System and In-Vehicle Communication System" has functions such as voice data collection, speech recognition, translation, confidential information protection, and real-time display. This system operates particularly effectively in multilingual environments and situations with a high risk of information leakage.

[0184] 1. Collection methods

[0185] The server uses microphones or digital audio recorders installed inside the vehicle or conference room as a means of collecting audio data. This allows for real-time capture of user speech.

[0186] 2. Transmission method

[0187] The collected audio data is transmitted to the server in real time, using a network interface and data transfer protocol. The server divides the received audio data into appropriate sizes and proceeds to the next processing step.

[0188] 3. Speech recognition means

[0189] The transmitted audio data is converted into text data by a speech recognition system that uses a generative AI model. Specifically, the Google Cloud Speech-to-Text API is used. This speech recognition system achieves high-precision speech-to-text conversion.

[0190] 4. Translation methods

[0191] The converted text data is translated into different languages ​​as needed using translation tools. This utilizes generative AI models such as the Google Cloud Translate API. This facilitates smooth communication between languages.

[0192] 5. Filtering means

[0193] The translated text data is filtered to detect and mask or replace sensitive information. Regular expressions and custom algorithms are used to ensure that sensitive information is not leaked.

[0194] 6. Display means

[0195] The final text data is sent to the receiving terminal and displayed in real time. Passengers can check the spoken content and translation results in real time using the in-vehicle display. This display is processed using the Google Cloud Speech-to-Text API and the Google Cloud Translate API.

[0196] Examples of specific cases and prompt statements

[0197] For example, if you say "This project is a secret" in Japanese, it will be processed as follows:

[0198] 1. Audio data collection: The in-car microphone captures the phrase "This project is confidential."

[0199] 2. Speech Recognition: The Google Cloud Speech-to-Text API was used to convert the text to "This project is a secret."

[0200] 3. Translation: Translated to "This project is confidential" using the Google Cloud Translate API.

[0201] 4. Filtering: Detect the word "secret" and replace it with "[filtered]".

[0202] 5. Display: The in-vehicle display will show "This project is [filtered]".

[0203] Examples of prompt statements are as follows:

[0204] Please use the Google Cloud Speech-to-Text API to transcribe live meeting audio.

[0205] Please translate the Japanese text into English using the Google Cloud Translate API.

[0206] Detect the words "secret" and "confidential" in the text and replace them with "[filtered]".

[0207] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0208] Step 1:

[0209] Collection of audio data

[0210] Subject: terminal

[0211] Operation: When a user speaks inside an autonomous vehicle, their voice is captured in real time on the device.

[0212] Input: User's spoken voice

[0213] Data processing / data calculation: Audio data is captured in an appropriate format (e.g., LINEAR16).

[0214] Output: Audio data file (byte data)

[0215] Step 2:

[0216] Sending audio data

[0217] Subject: terminal

[0218] Operation: Audio data captured on the device is sent to the server in real time via the network.

[0219] Input: Audio data file (byte data)

[0220] Data processing / data calculation: Audio data is divided into small chunks to make it easier to transmit over the network.

[0221] Output: Divided audio data chunks

[0222] Step 3:

[0223] Speech recognition

[0224] Subject: Server

[0225] Operation: Audio data sent to the server is converted into text data using a generative AI model.

[0226] Input: Split audio data chunks

[0227] Data processing / data calculation: Convert audio data to text data using the Google Cloud Speech-to-Text API.

[0228] Output: Text data

[0229] Step 4:

[0230] translation

[0231] Subject: Server

[0232] Operation: Text data is translated into different languages ​​as needed.

[0233] Input: Text data

[0234] Data processing / data calculation: Translate text data into a specified language using the Google Cloud Translate API.

[0235] Output: Translated text data

[0236] Step 5:

[0237] filtering

[0238] Subject: Server

[0239] Operation: Sensitive information is detected from translated text data and masked or replaced.

[0240] Input: Translated text data

[0241] Data processing / data calculations: Detect sensitive information using regular expressions and custom algorithms, and perform masking or replacement.

[0242] Output: Filtered text data

[0243] Step 6:

[0244] Sending text data

[0245] Subject: Server

[0246] Operation: The final text data is sent to the receiving terminal.

[0247] Input: Filtered text data

[0248] Data processing / data calculation: Encode data in the appropriate format and prepare it for transmission.

[0249] Output: Text data to be sent

[0250] Step 7:

[0251] Real-time display

[0252] Subject: terminal

[0253] Operation: The final text data is displayed in real time on the in-vehicle display.

[0254] Input: Submitted text data

[0255] Data processing / data calculation: Formatting text data for screen display.

[0256] Output: Text data displayed on the screen

[0257] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0258] This invention relates to a conferencing system that combines speech recognition, translation, confidential information protection, emotion recognition, and real-time captioning. The system aims to collect audio data, transmit it to a server, perform speech recognition to convert it into text data, translate and filter it as needed, recognize the user's emotions, and finally display it on the terminal in real time.

[0259] System Overview

[0260] This system is configured as follows: First, when a user speaks during a meeting, their voice is captured in real time by the terminal. The captured voice data is appropriately segmented and sent to the server. The server receives the voice data and converts it into text data using an AI speech recognition model. The converted text data is translated into different languages ​​by translation means as needed. Subsequently, sensitive information is detected and masked or replaced in the text data by filtering means. Furthermore, the emotion engine identifies the user's emotional state from the voice data, and this information is added to the text data. The final text data is sent back to the terminal and displayed to the user in real time.

[0261] Specific example

[0262] Example 1: Japanese meeting

[0263] Let's say User A says "The next project starts next month" during a Zoom meeting in Japanese. This audio is captured by the device, split into 1-second segments, and sent to the server. The server inputs this audio into an AI speech recognition model and converts it into text data: "The next project starts next month." This text data is checked by a filtering mechanism, but since it does not contain any particularly confidential information, it is sent back to the device as is. The emotion engine identifies User A's emotions (e.g., joy or anxiety) from the audio data and adds that information to the text data. The device displays this text data as real-time subtitles on the user's screen.

[0264] Example 2: A meeting requiring translation from Japanese to English

[0265] In another scenario, consider a situation where User B says in Japanese, "We have achieved our sales targets." This audio is also captured by the device, split, and sent to the server. The server inputs the audio into an AI speech recognition model and generates text data that reads, "We have achieved our sales targets." This text data is then translated into English by a translation tool, resulting in "We have achieved our sales targets." This translation is also checked by a filtering tool to ensure it does not contain any confidential information. The emotion engine then identifies User B's emotions (e.g., a sense of accomplishment or relief) from the audio data and adds this information to the text data. The final text data is sent to the device and displayed as English subtitles.

[0266] Program processing

[0267] 1. Audio data collection: The device captures the user's spoken audio in real time, divides it into appropriate-sized chunks, and sends them to the server.

[0268] 2. Speech Recognition: The server receives the speech data and inputs it into the AI ​​speech recognition model to convert it into text data.

[0269] 3. Translation: If necessary, the converted text data is input into a generated AI translation model and translated into the specified language.

[0270] 4. Filtering: The server detects and masks or replaces confidential company information within the text data.

[0271] 5. Emotion Recognition: The emotion engine recognizes the user's emotions from the audio data and adds that emotion information to the text data.

[0272] 6. Subtitle Display: The final text data is sent to the device and displayed on the user's screen in real time. This allows the user to see subtitles that include the words spoken during the meeting and the emotions they conveyed.

[0273] In this way, the present invention realizes a system that accurately recognizes the speech during a meeting, translates it as needed, appropriately protects confidential information, and displays subtitles in real time including the user's emotional information.

[0274] The following describes the processing flow.

[0275] Step 1:

[0276] Collection of audio data

[0277] User: Speak during the meeting.

[0278] Terminal: To capture the user's speech in real time, obtain the audio stream using the API of the meeting system. Divide the obtained audio data into a certain length (for example, every 1 second).

[0279] Terminal: Store the divided audio data chunks in a buffer and prepare to send them to the server.

[0280] Step 2:

[0281] Transmission of audio data

[0282] Terminal: Send the audio data chunks stored in the buffer to the server at an appropriate timing. Be aware of minimizing the delay considering the network situation.

[0283] Step 3:

[0284] Speech recognition

[0285] Server: Receive the audio data chunks sent from the terminal.

[0286] Server: Input the received audio data chunk into an AI speech recognition model (e.g., Google Speech-to-Text API, etc.) to generate corresponding text data.

[0287] Server: Perform post-processing such as noise removal and string normalization on the text data obtained from the speech recognition model to improve accuracy.

[0288] Step 4:

[0289] Translation (if necessary)

[0290] Server: Compare the language in which the text data was generated with the display language specified by the user to determine whether translation is necessary.

[0291] Server: If translation is necessary, input the text data into an AI translation model (e.g., Google Translate API, etc.) and translate it into the specified language.

[0292] Server: Check the translation result and perform fine-tuning if necessary.

[0293] Step 5:

[0294] Filtering of outside-company confidential information

[0295] Server: Use filtering means to detect outside-company confidential information in the generated text data (including the case after translation).

[0296] Server: Mask the detected outside-company confidential information or replace it with appropriate tags. This prevents the leakage of confidential information.

[0297] Server: Generate the final filtered text data.

[0298] Step 6:

[0299] Emotion recognition

[0300] Server: Input voice data into the emotion engine to identify the user's emotional state.

[0301] Server: Add the identified emotion information to the text data. At this time, clarify which speech the emotion information specifically corresponds to.

[0302] Step 7:

[0303] Transmission and display of subtitle data

[0304] Server: Prepare to transmit the final text data to the terminal.

[0305] Server: Transmit the final text data to the terminal.

[0306] Terminal: Receive the text data transmitted from the server.

[0307] Terminal: Display the received text data on the user's screen in real time. As a result, the user can view the speech during the meeting as subtitles and at the same time read the emotions of the speaker.

[0308] Through this series of processing flows, this system can achieve high-precision speech recognition and translation, and can provide subtitles in real time including the user's emotion information while appropriately protecting confidential information. [[ID=3*]]

[0309] (Example 2)

[0310] Next, Example 2 will be described. In the following description, the data processing device 12 is referred to as the "server", and the smart device 14 is referred to as the "terminal".

[0311] Conventional conferencing systems often handle speech recognition, translation, and confidential information protection separately, lacking a system that integrates and processes all of these in real time. Furthermore, the lack of a function to recognize and display user emotions can lead to a decline in communication quality. To address these problems, the present invention aims to provide a real-time captioning system that combines speech recognition, translation, confidential information protection, and emotion recognition.

[0312] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0313] In this invention, the server includes speech recognition means, translation means, filtering means, emotion recognition means, and means for transmitting the final text data to a receiving terminal and displaying it in real time. This makes it possible to recognize speech during a meeting in real time, translate it as needed, and display subtitles in real time, including the user's emotion information, while appropriately protecting confidential information.

[0314] "Audio data" refers to information that represents audio signals in digital format, capturing the content of the user's speech.

[0315] "Collection means" refers to a device or software for collecting voice data spoken by a user.

[0316] "Transmission means" refers to a device or software for transmitting collected audio data to a server in real time.

[0317] "Speech recognition means" refers to a device or software for converting speech data into text data.

[0318] "Translation means" refers to a device or software for translating converted text data into a different language.

[0319] "Filtering means" refers to a device or software for detecting, masking, or replacing confidential information from translated text data.

[0320] "Emotion recognition means" refers to a device or software that identifies a user's emotions from audio data and adds that information to text data.

[0321] "Display means" refers to a device or software that transmits the final text data to a receiving terminal and displays it on the user's screen in real time.

[0322] The system of this invention collects audio data spoken during a meeting in real time and performs speech recognition, translation, confidential information protection, sentiment recognition, and real-time subtitle display. This system consists of a server, terminals, and users.

[0323] First, the device captures the user's speech in real time. This capture uses an audio input device (typically a microphone). For example, if the user is using conferencing software such as Zoom or Teams, the audio capture function built into that application is used. The captured audio data is divided into appropriate chunk sizes and sent to the server.

[0324] The server feeds the received audio data into a speech recognition system. This speech recognition system uses a generative artificial intelligence model such as the Google Speech-to-Text API to convert the audio data into highly accurate text data. The converted text data is then sent to a translation system as needed to be translated into the specified language. For example, using the DeepL API, the Japanese text "We have achieved our sales targets" can be translated into the English text "We have achieved our sales targets".

[0325] Next, the server uses filtering mechanisms to detect sensitive information within the converted and translated text data. Using the Data Loss Prevention API, etc., if sensitive information is found, that portion is masked or replaced. For example, if "sales target" is determined to be sensitive information, it will be masked to something like "We achieved our target."

[0326] Furthermore, the server uses emotion recognition to identify the user's emotions from the audio data and adds that information to the text data. Google Cloud's Natural Language API, among others, is used for this emotion recognition. For example, if the user's emotions (such as joy or accomplishment) are detected from the tone and pitch of their voice, that emotion information is added to the text data as a label.

[0327] Once the final text data is generated, the server sends it back to the terminal. The terminal displays the returned text data on the user's screen in real time. This allows the user to see subtitles of what was said during the meeting, including the emotions expressed. For example, it might display something like, "The next project starts next month [excited]."

[0328] Specific example

[0329] If user A says "The next project starts next month" during a Zoom meeting, this audio is captured on the device, split, and sent to the server. The server feeds this audio into the Google Speech-to-Text API to generate text data that reads "The next project starts next month." This data is then checked by the Data Loss Prevention API and determined to be free of confidential information. Furthermore, an emotion recognition engine adds emotion information, such as "joy." This final text data is then sent to the device and displayed in real time.

[0330] In another scenario, User B says in Japanese, "We have achieved our sales targets." This audio is captured on the device, split, and sent to the server. The server uses the Google Speech-to-Text API and the DeepL API to translate this audio into the English text, "We have achieved our sales targets." The Data Loss Prevention API then determines that no sensitive information is included. An emotion recognition engine adds the emotion "sense of accomplishment." The resulting text data is sent to the device and displayed in real time.

[0331] Example of a prompt

[0332] Speech recognition prompt:

[0333] Please convert "The next project will start next month" into text.

[0334] Translation prompt:

[0335] Please translate "We achieved our sales target" into English.

[0336] Emotion recognition prompt:

[0337] Identify the user's emotions from the audio data and add annotations to the results.

[0338] Thus, the system of the present invention improves user communication by recognizing, translating, filtering, and recognizing the emotions of speech during meetings in real time, and displaying it as subtitles.

[0339] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0340] Specific flow of the system program's processing

[0341] Step 1:

[0342] Collection of audio data

[0343] The device uses an audio input device (microphone) to capture the user's spoken audio in real time. For example, if a user says "The next project starts next month" in a Zoom meeting, this audio will be captured.

[0344] Input: User's spoken audio.

[0345] Specific operation: The captured audio data is divided into chunks of one second each. The terminal then sends this chunked audio data to the next processing step.

[0346] Step 2:

[0347] Sending audio data

[0348] The terminal sends chunked audio data to the server in real time.

[0349] Input: Audio data chunked every second.

[0350] Specific operation: The device uploads this data to the server via the internet. For example, real-time transfer using WebSocket is a possible approach.

[0351] Step 3:

[0352] Speech recognition

[0353] The server inputs the received audio data into an AI speech recognition model and converts it into text data. APIs such as Google Speech-to-Text are used for this purpose.

[0354] Input: Audio data sent from the device.

[0355] Output: Converted text data. (Example: "The next project starts next month.")

[0356] Specific operation: The server processes the audio chunks sequentially to generate high-precision text data. This data is then sent to subsequent processing steps.

[0357] Step 4:

[0358] translation

[0359] The server feeds the generated text data into the translation system as needed, and translates it into the desired language. The DeepL API is used for translation.

[0360] Input: Converted text data. (Example: "The next project starts next month.")

[0361] Output: Translated text data. (Example: "The next project will start from next month")

[0362] Specific operation: The server sends text data based on the configured target language and receives the translation result. This data is then sent to the next filtering step.

[0363] Step 5:

[0364] filtering

[0365] The server filters the translated text data and takes measures to detect sensitive information. Data Loss Prevention APIs, among others, are used.

[0366] Input: Translated text data. (Example: "The next project will start from next month")

[0367] Output: Filtered text data. (If there is confidential information, e.g., "The next project will start from next month")

[0368] Specific operation: The server scans the text data and detects keywords related to sensitive information. Those parts are then replaced or masked.

[0369] Step 6:

[0370] emotion recognition

[0371] The server uses emotion recognition to identify the user's emotions from the audio data and adds that information to the text data. Google Cloud's Natural Language API is used.

[0372] Input: User's voice data and filtered text data. (Example: "The next project will from next month")

[0373] Output: Text data with emotion labels. (Example: "The next project will from next month [joy]")

[0374] Specific operation: Emotional features such as tone and pitch are extracted from audio data, and the detected emotions (joy, anxiety, etc.) are added as labels to the text data.

[0375] Step 7:

[0376] Display subtitles

[0377] The server sends the final text data to the terminal and displays it on the user's screen in real time.

[0378] Input: Text data with emotion labels attached. (Example: "The next project will from next month [joy]")

[0379] Output: Subtitles displayed on the user's device. (Example: "The next project will be from next month [joy]")

[0380] Specific operation: The device displays this data in the caption box of the conferencing software. This allows users to see the content and sentiment information of what is being said in real time.

[0381] This series of processing steps enables a system that recognizes, translates, filters, and recognizes the sentiment of audio data during a meeting in real time, and displays it to the user immediately.

[0382] (Application Example 2)

[0383] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0384] Traditional meeting and customer service systems had limited voice recognition and translation capabilities, making them inadequate for real-time communication, multilingual support, and the protection of confidential information. Furthermore, the lack of means to recognize and provide real-time information about the emotions of customers and meeting participants resulted in a decline in communication quality. Additionally, the reliability of translation results and the protection of confidential information were insufficient, necessitating appropriate solutions.

[0385] In Application Example 2, the specific processing performed by the specific processing unit 290 of the data processing device 12 is realized by the following means. In this invention, the server includes means for collecting voice data, means for transmitting the collected voice data to the server in real time, voice recognition means for converting the transmitted voice data into text data, translation means for translating the converted text data into different languages, filtering means for detecting, masking, or replacing confidential information from the translated text data, emotion recognition means for identifying emotional states from the voice data and adding that information to the text data, and means for transmitting the final text data to a receiving terminal and displaying it in real time. This enables voice recognition, translation, protection of confidential information, and real-time display including emotion recognition.

[0386] "Audio data" refers to a data format in which sound is recorded electronically and can be analyzed and converted through computer processing.

[0387] "Means of collection" refers to devices or systems that capture audio data in real time and store it in digital format.

[0388] "Means of sending to the server in real time" refers to communication methods for transferring collected audio data to the server without delay.

[0389] "Speech recognition means" refers to a software or hardware configuration for analyzing speech data and converting it into text data.

[0390] "Translation means" refers to software or services for converting text data into different languages.

[0391] A "filtering method" is an algorithm that automatically detects confidential information from text data and masks or replaces that portion.

[0392] "Emotion recognition means" refers to software or models that detect emotions from audio or text data and add that information.

[0393] "Final text data" refers to the character data after processing is complete, including all processing such as speech recognition, translation, filtering, and sentiment recognition.

[0394] A "receiving terminal" is a device used to display the final text data, and includes smartphones, smart glasses, and other similar devices.

[0395] The embodiments for carrying out the present invention will be described in detail below.

[0396] This system processes voice data collection, speech recognition, translation, filtering, sentiment recognition, and real-time display as a series of processes. The voice spoken by the user is first captured in real time by a dedicated device (such as smart glasses or a smartphone). Suitable hardware includes a microphone and a high-speed central processing unit (CPU).

[0397] After the collection of audio data is complete, the collected data is transmitted to the server in real time. High-speed communication lines such as Wi-Fi or 4G / 5G are used for this transmission. The server first appropriately divides (chunks) the received audio data into digital format, and then divides it into appropriate sizes according to the amount of data.

[0398] The segmented audio data is then converted into text data using speech recognition software such as the Google Cloud Speech-to-Text API or IBM Watson® Speech to Text. The converted text data is then translated into the specified language using the Google Cloud Translation API, if necessary.

[0399] The translated text data is further filtered to automatically detect and mask or replace personal or confidential information. This minimizes the risk of information leakage.

[0400] Subsequently, IBM Watson Tone Analyzer or a similar emotion recognition engine is used to identify the user's emotional state from the text data. Emotional information is added to the text data, and the final text data including this information is generated.

[0401] This final text data is sent back to the device and displayed to the user in real time. By displaying it on a screen such as smart glasses or a smartphone, the user can see not only what they have said, but also the emotions and important information contained within that statement in real time.

[0402] Specific example:

[0403] The following prompt messages are a concrete example of how this system can be applied to a customer service system in a physical store.

[0404] "Development of a system that recognizes emotions from customer statements, performs necessary translations, and displays them in real time on smart glasses."

[0405] Follow these steps to develop a system that recognizes emotions from customer speech, performs necessary translations, and displays them in real time on smart glasses.

[0406] 1. Capture customer voices in real time and convert them to text.

[0407] 2. If necessary, translate the text into the specified language.

[0408] 3. Detect and mask confidential information from the text.

[0409] 4. Recognize customer emotions from voice and add that information to text.

[0410] 5. Display the final text data on the smart glasses' display in real time.

[0411] This system facilitates multilingual support and enables more effective customer service by recognizing customer emotions. Furthermore, it provides a user-friendly environment while appropriately protecting confidential information.

[0412] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0413] Step 1:

[0414] The device captures the user's spoken voice in real time and converts it into a digital format. Specifically, the device's microphone collects the voice and saves it as an audio file. The input to this process is the user's voice data, and the output is a digital audio file.

[0415] Step 2:

[0416] The terminal transmits captured audio data to the server in real time. Specifically, the terminal uses a high-speed communication line such as Wi-Fi or 4G / 5G to divide the audio data into data packets and send them to the server. The input to this process is an audio file, and the output is the audio data sent to the server.

[0417] Step 3:

[0418] The server divides the received audio data into appropriate-sized chunks. Specifically, the server divides the audio file at regular time intervals (e.g., 1 second) and saves each chunk separately. The input to this process is the transmitted audio data, and the output is the divided audio chunks.

[0419] Step 4:

[0420] The server inputs the divided audio chunks into a speech recognition system and converts them into text data. Specifically, the server calls the Google Cloud Speech-to-Text API to convert the audio data into text data. The input to this process is audio chunks, and the output is text data.

[0421] Step 5:

[0422] The server translates the converted text data into a different language. Specifically, the server uses the Google Cloud Translation API to translate the text data into the specified target language. The input to this process is the text data, and the output is the translated text data.

[0423] Step 6:

[0424] The server detects sensitive information from translated text data and masks or replaces it. Specifically, the server uses regular expressions and machine learning models to identify personal and sensitive information and replaces the detected information with masking characters such as "". The input to this process is translated text data, and the output is filtered text data.

[0425] Step 7:

[0426] The server inputs filtered text data into an emotion recognition system, identifies the user's emotional state, and adds that information to the text data. Specifically, the server uses IBM Watson Tone Analyzer to perform emotion analysis and adds the results to the original text data. The input to this process is filtered text data, and the output is text data with added emotion information.

[0427] Step 8:

[0428] The server sends the final text data to the receiving terminal and displays it in real time. Specifically, the server sends the generated text data to the terminal as real-time subtitles, which the terminal displays on its screen. The input to this process is text data with added emotional information, and the output is real-time subtitles displayed on the user's screen.

[0429] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0430] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0431] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0432] [Second Embodiment]

[0433] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0434] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0435] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0436] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0437] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0438] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0439] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0440] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0441] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0442] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0443] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0444] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0445] This invention relates to a system that combines speech recognition, translation, confidential information protection, and real-time captioning in a conference system. The system aims to collect audio data, transmit it to a server, perform speech recognition to convert it into text data, translate and filter it as needed, and finally display it on a terminal in real time.

[0446] System Overview

[0447] This system is configured as follows: First, when a user speaks during a meeting, their voice is captured in real time by the terminal. The captured voice data is appropriately segmented and sent to the server. The server receives the voice data and converts it into text data using an AI speech recognition model. The converted text data is translated into different languages ​​by translation means as needed. Subsequently, sensitive information is detected and masked or replaced in the text data by filtering means. The final text data is sent back to the terminal and displayed to the user in real time.

[0448] Specific example

[0449] Example 1: Japanese meeting

[0450] Let's say User A says "The next project starts next month" during a Zoom meeting in Japanese. This audio is captured by the device, split into 1-second segments, and sent to the server. The server inputs this audio into an AI speech recognition model and converts it into text data: "The next project starts next month." This text data is checked by a filtering mechanism, but since it does not contain any particularly confidential information, it is sent back to the device as is. The device displays this text data as real-time subtitles on the user's screen.

[0451] Example 2: A meeting requiring translation from Japanese to English

[0452] In another scenario, consider a situation where user B says in Japanese, "We have achieved our sales targets." This audio is also captured by the terminal, split, and sent to the server. The server inputs the audio into an AI speech recognition model and generates text data that reads, "We have achieved our sales targets." This text data is then translated into English by a translation tool, resulting in "We have achieved our sales targets." This translation is also checked by a filtering tool and determined not to contain any confidential information. The final text data is then sent to the terminal and displayed as English subtitles.

[0453] Program processing

[0454] 1. Audio data collection: The device captures the user's spoken audio in real time, divides it into appropriate-sized chunks, and sends them to the server.

[0455] 2. Speech Recognition: The server receives the speech data and inputs it into the AI ​​speech recognition model to convert it into text data.

[0456] 3. Translation: If necessary, the converted text data is input into a generated AI translation model and translated into the specified language.

[0457] 4. Filtering: The server detects and masks or replaces confidential company information within the text data.

[0458] 5. Subtitle display: The final text data is sent to the device and displayed on the user's screen in real time.

[0459] In this way, the present invention realizes a system that accurately recognizes speech during meetings, translates it as needed, and displays subtitles in real time while protecting confidential information.

[0460] The following describes the processing flow.

[0461] Step 1:

[0462] Collection of audio data

[0463] User: Speaks during the meeting.

[0464] Terminal: To capture user speech in real time, the system uses the conferencing system's API to obtain the audio stream. The acquired audio data is then divided into segments of a fixed length (e.g., every second).

[0465] Terminal: Stores the divided audio data chunks in a buffer and prepares them for transmission to the server.

[0466] Step 2:

[0467] Sending audio data

[0468] Terminal: Sends audio data chunks stored in the buffer to the server at the appropriate time. Considering network conditions, it is mindful of minimizing latency.

[0469] Step 3:

[0470] Speech recognition

[0471] Server: Receives audio data chunks sent from the terminal.

[0472] Server: Inputs the received audio data chunks into an AI speech recognition model (e.g., Google Speech-to-Text API) to generate the corresponding text data.

[0473] Server: The server processes the text data obtained from the speech recognition model, performing post-processing such as noise reduction and string normalization to improve accuracy.

[0474] Step 4:

[0475] Translation (if necessary)

[0476] Server: Compares the language in which the text data was generated with the display language specified by the user to determine whether translation is necessary.

[0477] Server: When translation is needed, it inputs text data into an AI translation model (e.g., Google Translate API) and translates it into the specified language.

[0478] Server: Review the translation results and make any necessary adjustments.

[0479] Step 5:

[0480] Filtering of confidential company information

[0481] Server: Uses filtering mechanisms to detect confidential company information within the generated text data (including after translation).

[0482] Server: Mask or replace detected confidential information with appropriate tags. This prevents the leakage of sensitive information.

[0483] Server: Generates the final filtered text data.

[0484] Step 6:

[0485] Subtitle data transmission and display

[0486] Server: Prepares to send the final text data to the terminal.

[0487] Server: Sends the final text data to the terminal.

[0488] Terminal: Receives text data sent from the server.

[0489] Terminal: Displays received text data on the user's screen in real time. This allows the user to see what is being said during the meeting as subtitles.

[0490] This processing flow enables the system to achieve highly accurate speech recognition and translation, providing users with real-time subtitles while appropriately protecting confidential information.

[0491] (Example 1)

[0492] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0493] In conferencing systems, there is a demand for real-time subtitle display while simultaneously supporting multiple languages ​​and protecting confidential information. However, current technology does not integrate the processes of speech recognition, translation, and filtering, making real-time processing difficult. Furthermore, if the efficient splitting and transmission of audio data are not performed properly, processing delays and reduced accuracy can occur.

[0494] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0495] In this invention, the server includes means for transmitting audio data to the server in real time, speech recognition means for converting the transmitted audio data into text data, translation means for translating the converted text data into different languages, filtering means for detecting, masking, or replacing confidential information from the translated text data, means for transmitting the final text data to a receiving terminal and displaying it in real time, and means for dividing the audio data into appropriate sizes. This enables real-time subtitle display in a conference system while achieving multilingual support and protection of confidential information.

[0496] "Audio data" refers to audio signals or digital data collected during meetings or conversations.

[0497] A "server" is a data processing device or computer system on a network that receives and processes audio data.

[0498] "Text data" refers to character information converted from speech data by speech recognition technology.

[0499] "Speech recognition means" refers to recognition technology or algorithms for converting speech data into text data.

[0500] "Translation methods" refer to techniques or algorithms that convert text data into different languages.

[0501] "Filtering means" refers to techniques or algorithms that detect and mask or replace confidential information from text data.

[0502] A "receiving terminal" is a device that receives the final text data sent from the server and displays it to the user.

[0503] "Means for displaying in real time" refers to a technology or device that displays text data that has undergone conversion and filtering processes without delay.

[0504] "Means for dividing into appropriate sizes" refers to techniques or devices for dividing audio data into appropriate time intervals so that it can be easily processed.

[0505] Modes for carrying out the invention

[0506] This invention relates to a conferencing system that combines speech recognition, translation, confidential information protection, and real-time captioning. This system uses specific hardware and software to collect, transmit, analyze, convert, filter, and display audio data.

[0507] Collection and transmission of audio data

[0508] When a user speaks during a meeting, the device captures their audio in real time. For example, Zoom or Microsoft Teams can be used as the meeting software. The captured audio data is divided into appropriate sizes (e.g., every second) and sent from the device to the server. It is desirable that the audio data be compressed during transmission.

[0509] Speech recognition

[0510] The server receives audio data sent from the terminal in real time. The audio data is input into an AI speech recognition model (e.g., Google Speech-to-Text API) and converted into text data. For example, the statement "The next project starts next month" is converted into the text data "The next project starts next month".

[0511] translation

[0512] The converted text data is then fed into translation models (e.g., DeepL, Google Translate) as needed, and translated into different languages. For example, the Japanese phrase "We have achieved our sales targets" is translated into English as "We have achieved our sales targets."

[0513] filtering

[0514] The server inspects the translated text data and detects sensitive information. Using natural language processing tools (e.g., spaCy, Microsoft Azure Text Analytics), it identifies specific sensitive information and masks or replaces the relevant sections. For example, "We achieved our sales target" might be masked to "Our sales target is..."

[0515] Subtitle display

[0516] The final text data is sent from the server to the terminal. The terminal displays the received text data as real-time subtitles on the user's screen. Subtitle display software such as OBS Studio is used for display.

[0517] Specific example

[0518] Example 1: Japanese meeting

[0519] User A: In a Japanese Zoom meeting, says, "The next project will start next month."

[0520] Terminal: Captures audio in real time, splits it into 1-second segments, and sends them to the server.

[0521] Server: Receives the audio and uses the Google Speech-to-Text API to convert it into text data that says, "The next project starts next month."

[0522] Server: Inspects the text data and, since it does not contain any particularly sensitive information, sends it to the terminal as is.

[0523] Terminal: Displays text data as real-time subtitles.

[0524] Example 2: A meeting requiring translation from Japanese to English

[0525] User B: Says in Japanese, "We have achieved our sales target."

[0526] Terminal: Captures audio and sends it to the server.

[0527] Server: Inputs the audio into an AI speech recognition model and generates text data that says, "Sales target achieved."

[0528] Server: Inputs text data into DeepL and obtains the English translation "We have achieved our sales targets".

[0529] Server: The translation results are checked and determined not to contain any confidential information.

[0530] Device: Display as English subtitles.

[0531] Example of a prompt

[0532] Speech Recognition and Real-Time Display for Japanese Meetings: "Please describe a program that converts Japanese audio during a meeting into text data and displays it in real time."

[0533] Translation and display from Japanese to English: "Please explain the process of translating Japanese audio into English and displaying it as subtitles in real time."

[0534] In this way, the present invention provides a means for efficiently and effectively conducting meetings by integrating speech recognition, translation, confidential information protection, and real-time captioning into a single system within a conference system.

[0535] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0536] Step 1:

[0537] Collection and capture of audio data

[0538] Input: Audio of a user speaking during a meeting.

[0539] Specific action: The user speaks through conferencing software such as Zoom or Teams.

[0540] Output: Audio data captured by the device.

[0541] Processing details: The device captures the user's speech in real time using the microphone. This audio data is saved in digital format and immediately split.

[0542] Step 2:

[0543] Splitting and transmitting audio data

[0544] Input: Captured audio data.

[0545] Specific operation: The device divides the captured audio data into 1-second segments. It also compresses the audio data to reduce unnecessary data.

[0546] Output: Split and compressed audio data.

[0547] Processing details: The terminal divides the captured audio data into appropriate chunks (every second), compresses the data, and sends it to the server. This process improves communication efficiency.

[0548] Step 3:

[0549] Receiving and recognizing audio data

[0550] Input: Split audio data sent from the device.

[0551] Specific operation: The server receives this data and passes it to an AI speech recognition model (e.g., Google Speech-to-Text API).

[0552] Output: Text data.

[0553] Processing details: The server inputs the received audio data into an AI speech recognition model and converts it into text data in real time. For example, "The next project starts next month" is generated as text data.

[0554] Step 4:

[0555] Text data translation

[0556] Input: Text data generated by speech recognition.

[0557] Specific operation: The server inputs text data into a translation model (e.g., Google Translate, DeepL) according to the requirements.

[0558] Output: Translated text data.

[0559] Processing details: The converted text data is input into a translation model and translated into the specified language. For example, "We have achieved our sales targets" is translated as "We have achieved our sales targets."

[0560] Step 5:

[0561] Filtering of translated text data

[0562] Input: Translated text data.

[0563] Specific operation: The server detects sensitive information within the text data using filtering tools (e.g., spaCy, Microsoft Azure Text Analytics).

[0564] Output: Text data with confidential information masked or replaced.

[0565] Processing details: Inspect the translated text data and mask any confidential information found (e.g., "Sales targets are") or replace it (e.g., "Top Secret Data" → "Important Data").

[0566] Step 6:

[0567] Sending the final text data to the receiving terminal

[0568] Input: Filtered text data.

[0569] Specific operation: The server sends the final text data to the receiving terminal.

[0570] Output: Text data sent to the receiving terminal.

[0571] Processing details: The final text data is sent from the server to the receiving terminal. The sent data is processed at the receiving terminal.

[0572] Step 7:

[0573] Real-time subtitle display

[0574] Input: The final text data sent from the server.

[0575] Specific operation: The receiving terminal displays this text data on its screen in real time. For example, subtitles can be displayed using OBS Studio or similar software.

[0576] Output: Real-time subtitles displayed on the user's screen.

[0577] Processing details: The receiving terminal displays the final text data as real-time subtitles. This makes it easier for users to understand the meeting content.

[0578] (Application Example 1)

[0579] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0580] Communication among passengers in autonomous vehicles presents challenges such as language barriers and the risk of confidential information leakage. Furthermore, while real-time speech recognition, translation, and the protection and display of confidential information are necessary, there is a lack of systems that can efficiently achieve these goals. This challenge is particularly pronounced in multilingual environments and business meetings where the risk of information leakage is high.

[0581] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0582] In this invention, the server includes means for collecting voice data, means for transmitting the collected voice data to the server in real time, and voice recognition means that uses a generative AI model to convert the voice data into text data. This enables support for communication between multiple languages ​​and protection of confidential information.

[0583] A "conference system and in-vehicle communication system" is a system that has functions such as voice data collection, speech recognition, translation, confidential information protection, and real-time display, and operates effectively in multilingual environments and situations with a high risk of information leakage.

[0584] "Means for collecting audio data" refers to devices such as microphones and digital audio recorders used to capture audio in real time from inside a vehicle or conference room.

[0585] "Means of sending to the server in real time" refers to network interfaces and data transfer protocols for sending captured audio data to the server without delay.

[0586] "Speech recognition means" refers to software or hardware used to convert speech data into text data, particularly those that utilize generative AI models.

[0587] "Translation means" refers to software or generative AI models used to translate text data obtained by speech recognition means into different languages.

[0588] "Filtering means" refers to algorithms or software used to detect confidential information from translated text data and to mask or replace that information.

[0589] "Means for sending to a receiving terminal and displaying in real time" refers to programs or hardware that send the final text data to a client device and display it on that device in real time.

[0590] "Means for displaying text data in real time on an in-vehicle display" refers to a device that has an interface function for displaying text data in real time on a screen or monitor installed inside a vehicle.

[0591] A "generative AI model" is an artificial intelligence algorithm trained to perform tasks such as speech recognition, translation, and sensitive information detection.

[0592] This invention is a system for supporting communication within autonomous vehicles and conference systems. The "Conference System and In-Vehicle Communication System" has functions such as voice data collection, speech recognition, translation, confidential information protection, and real-time display. This system operates particularly effectively in multilingual environments and situations with a high risk of information leakage.

[0593] 1. Collection methods

[0594] The server uses microphones or digital audio recorders installed inside the vehicle or conference room as a means of collecting audio data. This allows for real-time capture of user speech.

[0595] 2. Transmission method

[0596] The collected audio data is transmitted to the server in real time, using a network interface and data transfer protocol. The server divides the received audio data into appropriate sizes and proceeds to the next processing step.

[0597] 3. Speech recognition means

[0598] The transmitted audio data is converted into text data by a speech recognition system that uses a generative AI model. Specifically, the Google Cloud Speech-to-Text API is used. This speech recognition system achieves high-precision speech-to-text conversion.

[0599] 4. Translation methods

[0600] The converted text data is translated into different languages ​​as needed using translation tools. This utilizes generative AI models such as the Google Cloud Translate API. This facilitates smooth communication between languages.

[0601] 5. Filtering means

[0602] The translated text data is filtered to detect and mask or replace sensitive information. Regular expressions and custom algorithms are used to ensure that sensitive information is not leaked.

[0603] 6. Display means

[0604] The final text data is sent to the receiving terminal and displayed in real time. Passengers can check the spoken content and translation results in real time using the in-vehicle display. This display is processed using the Google Cloud Speech-to-Text API and the Google Cloud Translate API.

[0605] Examples of specific cases and prompt statements

[0606] For example, if you say "This project is a secret" in Japanese, it will be processed as follows:

[0607] 1. Audio data collection: The in-car microphone captures the phrase "This project is confidential."

[0608] 2. Speech Recognition: The Google Cloud Speech-to-Text API was used to convert the text to "This project is a secret."

[0609] 3. Translation: Translated to "This project is confidential" using the Google Cloud Translate API.

[0610] 4. Filtering: Detect the word "secret" and replace it with "[filtered]".

[0611] 5. Display: The in-vehicle display will show "This project is [filtered]".

[0612] Examples of prompt statements are as follows:

[0613] Please use the Google Cloud Speech-to-Text API to transcribe live meeting audio.

[0614] Please translate the Japanese text into English using the Google Cloud Translate API.

[0615] Detect the words "secret" and "confidential" in the text and replace them with "[filtered]".

[0616] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0617] Step 1:

[0618] Collection of audio data

[0619] Subject: terminal

[0620] Operation: When a user speaks inside an autonomous vehicle, their voice is captured in real time on the device.

[0621] Input: User's spoken voice

[0622] Data processing / data calculation: Audio data is captured in an appropriate format (e.g., LINEAR16).

[0623] Output: Audio data file (byte data)

[0624] Step 2:

[0625] Sending audio data

[0626] Subject: terminal

[0627] Operation: Audio data captured on the device is sent to the server in real time via the network.

[0628] Input: Audio data file (byte data)

[0629] Data processing / data calculation: Audio data is divided into small chunks to make it easier to transmit over the network.

[0630] Output: Divided audio data chunks

[0631] Step 3:

[0632] Speech recognition

[0633] Subject: Server

[0634] Operation: Audio data sent to the server is converted into text data using a generative AI model.

[0635] Input: Split audio data chunks

[0636] Data processing / data calculation: Convert audio data to text data using the Google Cloud Speech-to-Text API.

[0637] Output: Text data

[0638] Step 4:

[0639] translation

[0640] Subject: Server

[0641] Operation: Text data is translated into different languages ​​as needed.

[0642] Input: Text data

[0643] Data processing / data calculation: Translate text data into a specified language using the Google Cloud Translate API.

[0644] Output: Translated text data

[0645] Step 5:

[0646] filtering

[0647] Subject: Server

[0648] Operation: Sensitive information is detected from translated text data and masked or replaced.

[0649] Input: Translated text data

[0650] Data processing / data calculations: Detect sensitive information using regular expressions and custom algorithms, and perform masking or replacement.

[0651] Output: Filtered text data

[0652] Step 6:

[0653] Sending text data

[0654] Subject: Server

[0655] Operation: The final text data is sent to the receiving terminal.

[0656] Input: Filtered text data

[0657] Data processing / data calculation: Encode data in the appropriate format and prepare it for transmission.

[0658] Output: Text data to be sent

[0659] Step 7:

[0660] Real-time display

[0661] Subject: terminal

[0662] Operation: The final text data is displayed in real time on the in-vehicle display.

[0663] Input: Submitted text data

[0664] Data processing / data calculation: Formatting text data for screen display.

[0665] Output: Text data displayed on the screen

[0666] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0667] This invention relates to a conferencing system that combines speech recognition, translation, confidential information protection, emotion recognition, and real-time captioning. The system aims to collect audio data, transmit it to a server, perform speech recognition to convert it into text data, translate and filter it as needed, recognize the user's emotions, and finally display it on the terminal in real time.

[0668] System Overview

[0669] This system is configured as follows: First, when a user speaks during a meeting, their voice is captured in real time by the terminal. The captured voice data is appropriately segmented and sent to the server. The server receives the voice data and converts it into text data using an AI speech recognition model. The converted text data is translated into different languages ​​by translation means as needed. Subsequently, sensitive information is detected and masked or replaced in the text data by filtering means. Furthermore, the emotion engine identifies the user's emotional state from the voice data, and this information is added to the text data. The final text data is sent back to the terminal and displayed to the user in real time.

[0670] Specific example

[0671] Example 1: Japanese meeting

[0672] Let's say User A says "The next project starts next month" during a Zoom meeting in Japanese. This audio is captured by the device, split into 1-second segments, and sent to the server. The server inputs this audio into an AI speech recognition model and converts it into text data: "The next project starts next month." This text data is checked by a filtering mechanism, but since it does not contain any particularly confidential information, it is sent back to the device as is. The emotion engine identifies User A's emotions (e.g., joy or anxiety) from the audio data and adds that information to the text data. The device displays this text data as real-time subtitles on the user's screen.

[0673] Example 2: A meeting requiring translation from Japanese to English

[0674] In another scenario, consider a situation where User B says in Japanese, "We have achieved our sales targets." This audio is also captured by the device, split, and sent to the server. The server inputs the audio into an AI speech recognition model and generates text data that reads, "We have achieved our sales targets." This text data is then translated into English by a translation tool, resulting in "We have achieved our sales targets." This translation is also checked by a filtering tool to ensure it does not contain any confidential information. The emotion engine then identifies User B's emotions (e.g., a sense of accomplishment or relief) from the audio data and adds this information to the text data. The final text data is sent to the device and displayed as English subtitles.

[0675] Program processing

[0676] 1. Audio data collection: The device captures the user's spoken audio in real time, divides it into appropriate-sized chunks, and sends them to the server.

[0677] 2. Speech Recognition: The server receives the speech data and inputs it into the AI ​​speech recognition model to convert it into text data.

[0678] 3. Translation: If necessary, the converted text data is input into a generated AI translation model and translated into the specified language.

[0679] 4. Filtering: The server detects and masks or replaces confidential company information within the text data.

[0680] 5. Emotion Recognition: The emotion engine recognizes the user's emotions from the audio data and adds that emotion information to the text data.

[0681] 6. Subtitle Display: The final text data is sent to the device and displayed on the user's screen in real time. This allows the user to see subtitles that include the words spoken during the meeting and the emotions they conveyed.

[0682] In this way, the present invention realizes a system that accurately recognizes speech during meetings, translates it as needed, and displays subtitles in real time, including user sentiment information, while appropriately protecting confidential information.

[0683] The following describes the processing flow.

[0684] Step 1:

[0685] Collection of audio data

[0686] User: Speaks during the meeting.

[0687] Terminal: To capture user speech in real time, the system uses the conferencing system's API to obtain the audio stream. The acquired audio data is then divided into segments of a fixed length (e.g., every second).

[0688] Terminal: Stores the divided audio data chunks in a buffer and prepares them for transmission to the server.

[0689] Step 2:

[0690] Sending audio data

[0691] Terminal: Sends audio data chunks stored in the buffer to the server at the appropriate time. Considering network conditions, it is mindful of minimizing latency.

[0692] Step 3:

[0693] Speech recognition

[0694] Server: Receives audio data chunks sent from the terminal.

[0695] Server: Inputs the received audio data chunks into an AI speech recognition model (e.g., Google Speech-to-Text API) to generate the corresponding text data.

[0696] Server: The server processes the text data obtained from the speech recognition model, performing post-processing such as noise reduction and string normalization to improve accuracy.

[0697] Step 4:

[0698] Translation (if necessary)

[0699] Server: Compares the language in which the text data was generated with the display language specified by the user to determine whether translation is necessary.

[0700] Server: When translation is needed, it inputs text data into an AI translation model (e.g., Google Translate API) and translates it into the specified language.

[0701] Server: Review the translation results and make any necessary adjustments.

[0702] Step 5:

[0703] Filtering of confidential company information

[0704] Server: Uses filtering mechanisms to detect confidential company information within the generated text data (including after translation).

[0705] Server: Mask or replace detected confidential information with appropriate tags. This prevents the leakage of sensitive information.

[0706] Server: Generates the final filtered text data.

[0707] Step 6:

[0708] emotion recognition

[0709] Server: Inputs voice data into the emotion engine to identify the user's emotional state.

[0710] Server: Adds identified sentiment information to the text data. At this time, it clarifies which specific statement the sentiment information corresponds to.

[0711] Step 7:

[0712] Subtitle data transmission and display

[0713] Server: Prepares to send the final text data to the terminal.

[0714] Server: Sends the final text data to the terminal.

[0715] Terminal: Receives text data sent from the server.

[0716] Terminal: Displays received text data on the user's screen in real time. This allows the user to see what is being said in a meeting as subtitles and simultaneously understand the speaker's emotions.

[0717] This processing flow enables the system to achieve highly accurate speech recognition and translation, providing real-time subtitles that include user sentiment information while appropriately protecting confidential data.

[0718] (Example 2)

[0719] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0720] Conventional conferencing systems often handle speech recognition, translation, and confidential information protection separately, lacking a system that integrates and processes all of these in real time. Furthermore, the lack of a function to recognize and display user emotions can lead to a decline in communication quality. To address these problems, the present invention aims to provide a real-time captioning system that combines speech recognition, translation, confidential information protection, and emotion recognition.

[0721] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0722] In this invention, the server includes speech recognition means, translation means, filtering means, emotion recognition means, and means for transmitting the final text data to a receiving terminal and displaying it in real time. This makes it possible to recognize speech during a meeting in real time, translate it as needed, and display subtitles in real time, including the user's emotion information, while appropriately protecting confidential information.

[0723] "Audio data" refers to information that represents audio signals in digital format, capturing the content of the user's speech.

[0724] "Collection means" refers to a device or software for collecting voice data spoken by a user.

[0725] "Transmission means" refers to a device or software for transmitting collected audio data to a server in real time.

[0726] "Speech recognition means" refers to a device or software for converting speech data into text data.

[0727] "Translation means" refers to a device or software for translating converted text data into a different language.

[0728] "Filtering means" refers to a device or software for detecting, masking, or replacing confidential information from translated text data.

[0729] "Emotion recognition means" refers to a device or software that identifies a user's emotions from audio data and adds that information to text data.

[0730] "Display means" refers to a device or software that transmits the final text data to a receiving terminal and displays it on the user's screen in real time.

[0731] The system of this invention collects audio data spoken during a meeting in real time and performs speech recognition, translation, confidential information protection, sentiment recognition, and real-time subtitle display. This system consists of a server, terminals, and users.

[0732] First, the device captures the user's speech in real time. This capture uses an audio input device (typically a microphone). For example, if the user is using conferencing software such as Zoom or Teams, the audio capture function built into that application is used. The captured audio data is divided into appropriate chunk sizes and sent to the server.

[0733] The server feeds the received audio data into a speech recognition system. This speech recognition system uses a generative artificial intelligence model such as the Google Speech-to-Text API to convert the audio data into highly accurate text data. The converted text data is then sent to a translation system as needed to be translated into the specified language. For example, using the DeepL API, the Japanese text "We have achieved our sales targets" can be translated into the English text "We have achieved our sales targets".

[0734] Next, the server uses filtering mechanisms to detect sensitive information within the converted and translated text data. Using the Data Loss Prevention API, etc., if sensitive information is found, that portion is masked or replaced. For example, if "sales target" is determined to be sensitive information, it will be masked to something like "We achieved our target."

[0735] Furthermore, the server uses emotion recognition to identify the user's emotions from the audio data and adds that information to the text data. Google Cloud's Natural Language API, among others, is used for this emotion recognition. For example, if the user's emotions (such as joy or accomplishment) are detected from the tone and pitch of their voice, that emotion information is added to the text data as a label.

[0736] Once the final text data is generated, the server sends it back to the terminal. The terminal displays the returned text data on the user's screen in real time. This allows the user to see subtitles of what was said during the meeting, including the emotions expressed. For example, it might display something like, "The next project starts next month [excited]."

[0737] Specific example

[0738] If user A says "The next project starts next month" during a Zoom meeting, this audio is captured on the device, split, and sent to the server. The server feeds this audio into the Google Speech-to-Text API to generate text data that reads "The next project starts next month." This data is then checked by the Data Loss Prevention API and determined to be free of confidential information. Furthermore, an emotion recognition engine adds emotion information, such as "joy." This final text data is then sent to the device and displayed in real time.

[0739] In another scenario, User B says in Japanese, "We have achieved our sales targets." This audio is captured on the device, split, and sent to the server. The server uses the Google Speech-to-Text API and the DeepL API to translate this audio into the English text, "We have achieved our sales targets." The Data Loss Prevention API then determines that no sensitive information is included. An emotion recognition engine adds the emotion "sense of accomplishment." The resulting text data is sent to the device and displayed in real time.

[0740] Example of a prompt

[0741] Speech recognition prompt:

[0742] Please convert "The next project will start next month" into text.

[0743] Translation prompt:

[0744] Please translate "We achieved our sales target" into English.

[0745] Emotion recognition prompt:

[0746] Identify the user's emotions from the audio data and add annotations to the results.

[0747] Thus, the system of the present invention improves user communication by recognizing, translating, filtering, and recognizing the emotions of speech during meetings in real time, and displaying it as subtitles.

[0748] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0749] Specific flow of the system program's processing

[0750] Step 1:

[0751] Collection of audio data

[0752] The device uses an audio input device (microphone) to capture the user's spoken audio in real time. For example, if a user says "The next project starts next month" in a Zoom meeting, this audio will be captured.

[0753] Input: User's spoken audio.

[0754] Specific operation: The captured audio data is divided into chunks of one second each. The terminal then sends this chunked audio data to the next processing step.

[0755] Step 2:

[0756] Sending audio data

[0757] The terminal sends chunked audio data to the server in real time.

[0758] Input: Audio data chunked every second.

[0759] Specific operation: The device uploads this data to the server via the internet. For example, real-time transfer using WebSocket is a possible approach.

[0760] Step 3:

[0761] Speech recognition

[0762] The server inputs the received audio data into an AI speech recognition model and converts it into text data. APIs such as Google Speech-to-Text are used for this purpose.

[0763] Input: Audio data sent from the device.

[0764] Output: Converted text data. (Example: "The next project starts next month.")

[0765] Specific operation: The server processes the audio chunks sequentially to generate high-precision text data. This data is then sent to subsequent processing steps.

[0766] Step 4:

[0767] translation

[0768] The server feeds the generated text data into the translation system as needed, and translates it into the desired language. The DeepL API is used for translation.

[0769] Input: Converted text data. (Example: "The next project starts next month.")

[0770] Output: Translated text data. (Example: "The next project will start from next month")

[0771] Specific operation: The server sends text data based on the configured target language and receives the translation result. This data is then sent to the next filtering step.

[0772] Step 5:

[0773] filtering

[0774] The server filters the translated text data and takes measures to detect sensitive information. Data Loss Prevention APIs, among others, are used.

[0775] Input: Translated text data. (Example: "The next project will start from next month")

[0776] Output: Filtered text data. (If there is confidential information, e.g., "The next project will start from next month")

[0777] Specific operation: The server scans the text data and detects keywords related to sensitive information. Those parts are then replaced or masked.

[0778] Step 6:

[0779] emotion recognition

[0780] The server uses emotion recognition to identify the user's emotions from the audio data and adds that information to the text data. Google Cloud's Natural Language API is used.

[0781] Input: User's voice data and filtered text data. (Example: "The next project will from next month")

[0782] Output: Text data with emotion labels. (Example: "The next project will from next month [joy]")

[0783] Specific operation: Emotional features such as tone and pitch are extracted from audio data, and the detected emotions (joy, anxiety, etc.) are added as labels to the text data.

[0784] Step 7:

[0785] Display subtitles

[0786] The server sends the final text data to the terminal and displays it on the user's screen in real time.

[0787] Input: Text data with emotion labels attached. (Example: "The next project will from next month [joy]")

[0788] Output: Subtitles displayed on the user's device. (Example: "The next project will be from next month [joy]")

[0789] Specific operation: The device displays this data in the caption box of the conferencing software. This allows users to see the content and sentiment information of what is being said in real time.

[0790] This series of processing steps enables a system that recognizes, translates, filters, and recognizes the sentiment of audio data during a meeting in real time, and displays it to the user immediately.

[0791] (Application Example 2)

[0792] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0793] Traditional meeting and customer service systems had limited voice recognition and translation capabilities, making them inadequate for real-time communication, multilingual support, and the protection of confidential information. Furthermore, the lack of means to recognize and provide real-time information about the emotions of customers and meeting participants resulted in a decline in communication quality. Additionally, the reliability of translation results and the protection of confidential information were insufficient, necessitating appropriate solutions.

[0794] In Application Example 2, the specific processing performed by the specific processing unit 290 of the data processing device 12 is realized by the following means. In this invention, the server includes means for collecting voice data, means for transmitting the collected voice data to the server in real time, voice recognition means for converting the transmitted voice data into text data, translation means for translating the converted text data into different languages, filtering means for detecting, masking, or replacing confidential information from the translated text data, emotion recognition means for identifying emotional states from the voice data and adding that information to the text data, and means for transmitting the final text data to a receiving terminal and displaying it in real time. This enables voice recognition, translation, protection of confidential information, and real-time display including emotion recognition.

[0795] "Audio data" refers to a data format in which sound is recorded electronically and can be analyzed and converted through computer processing.

[0796] "Means of collection" refers to devices or systems that capture audio data in real time and store it in digital format.

[0797] "Means of sending to the server in real time" refers to communication methods for transferring collected audio data to the server without delay.

[0798] "Speech recognition means" refers to a software or hardware configuration for analyzing speech data and converting it into text data.

[0799] "Translation means" refers to software or services for converting text data into different languages.

[0800] A "filtering method" is an algorithm that automatically detects confidential information from text data and masks or replaces that portion.

[0801] "Emotion recognition means" refers to software or models that detect emotions from audio or text data and add that information.

[0802] "Final text data" refers to the character data after processing is complete, including all processing such as speech recognition, translation, filtering, and sentiment recognition.

[0803] A "receiving terminal" is a device used to display the final text data, and includes smartphones, smart glasses, and other similar devices.

[0804] The embodiments for carrying out the present invention will be described in detail below.

[0805] This system processes voice data collection, speech recognition, translation, filtering, sentiment recognition, and real-time display as a series of processes. The voice spoken by the user is first captured in real time by a dedicated device (such as smart glasses or a smartphone). Suitable hardware includes a microphone and a high-speed central processing unit (CPU).

[0806] After the collection of audio data is complete, the collected data is transmitted to the server in real time. High-speed communication lines such as Wi-Fi or 4G / 5G are used for this transmission. The server first appropriately divides (chunks) the received audio data into digital format, and then divides it into appropriate sizes according to the amount of data.

[0807] The segmented audio data is then converted into text data using speech recognition software such as the Google Cloud Speech-to-Text API or IBM Watson Speech to Text. The converted text data is then translated into the specified language using the Google Cloud Translation API, if necessary.

[0808] The translated text data is further filtered to automatically detect and mask or replace personal or confidential information. This minimizes the risk of information leakage.

[0809] Subsequently, IBM Watson Tone Analyzer or a similar emotion recognition engine is used to identify the user's emotional state from the text data. Emotional information is added to the text data, and the final text data including this information is generated.

[0810] This final text data is sent back to the device and displayed to the user in real time. By displaying it on a screen such as smart glasses or a smartphone, the user can see not only what they have said, but also the emotions and important information contained within that statement in real time.

[0811] Specific example:

[0812] The following prompt messages are a concrete example of how this system can be applied to a customer service system in a physical store.

[0813] "Development of a system that recognizes emotions from customer statements, performs necessary translations, and displays them in real time on smart glasses."

[0814] Follow these steps to develop a system that recognizes emotions from customer speech, performs necessary translations, and displays them in real time on smart glasses.

[0815] 1. Capture customer voices in real time and convert them to text.

[0816] 2. If necessary, translate the text into the specified language.

[0817] 3. Detect and mask confidential information from the text.

[0818] 4. Recognize customer emotions from voice and add that information to text.

[0819] 5. Display the final text data on the smart glasses' display in real time.

[0820] This system facilitates multilingual support and enables more effective customer service by recognizing customer emotions. Furthermore, it provides a user-friendly environment while appropriately protecting confidential information.

[0821] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0822] Step 1:

[0823] The device captures the user's spoken voice in real time and converts it into a digital format. Specifically, the device's microphone collects the voice and saves it as an audio file. The input to this process is the user's voice data, and the output is a digital audio file.

[0824] Step 2:

[0825] The terminal transmits captured audio data to the server in real time. Specifically, the terminal uses a high-speed communication line such as Wi-Fi or 4G / 5G to divide the audio data into data packets and send them to the server. The input to this process is an audio file, and the output is the audio data sent to the server.

[0826] Step 3:

[0827] The server divides the received audio data into appropriate-sized chunks. Specifically, the server divides the audio file at regular time intervals (e.g., 1 second) and saves each chunk separately. The input to this process is the transmitted audio data, and the output is the divided audio chunks.

[0828] Step 4:

[0829] The server inputs the divided audio chunks into a speech recognition system and converts them into text data. Specifically, the server calls the Google Cloud Speech-to-Text API to convert the audio data into text data. The input to this process is audio chunks, and the output is text data.

[0830] Step 5:

[0831] The server translates the converted text data into a different language. Specifically, the server uses the Google Cloud Translation API to translate the text data into the specified target language. The input to this process is the text data, and the output is the translated text data.

[0832] Step 6:

[0833] The server detects sensitive information from translated text data and masks or replaces it. Specifically, the server uses regular expressions and machine learning models to identify personal and sensitive information and replaces the detected information with masking characters such as "". The input to this process is translated text data, and the output is filtered text data.

[0834] Step 7:

[0835] The server inputs filtered text data into an emotion recognition system, identifies the user's emotional state, and adds that information to the text data. Specifically, the server uses IBM Watson Tone Analyzer to perform emotion analysis and adds the results to the original text data. The input to this process is filtered text data, and the output is text data with added emotion information.

[0836] Step 8:

[0837] The server sends the final text data to the receiving terminal and displays it in real time. Specifically, the server sends the generated text data to the terminal as real-time subtitles, which the terminal displays on its screen. The input to this process is text data with added emotional information, and the output is real-time subtitles displayed on the user's screen.

[0838] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0839] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0840] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0841] [Third Embodiment]

[0842] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0843] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0844] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0845] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0846] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0847] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0848] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0849] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0850] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0851] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0852] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0853] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0854] This invention relates to a system that combines speech recognition, translation, confidential information protection, and real-time captioning in a conference system. The system aims to collect audio data, transmit it to a server, perform speech recognition to convert it into text data, translate and filter it as needed, and finally display it on a terminal in real time.

[0855] System Overview

[0856] This system is configured as follows: First, when a user speaks during a meeting, their voice is captured in real time by the terminal. The captured voice data is appropriately segmented and sent to the server. The server receives the voice data and converts it into text data using an AI speech recognition model. The converted text data is translated into different languages ​​by translation means as needed. Subsequently, sensitive information is detected and masked or replaced in the text data by filtering means. The final text data is sent back to the terminal and displayed to the user in real time.

[0857] Specific example

[0858] Example 1: Japanese meeting

[0859] Let's say User A says "The next project starts next month" during a Zoom meeting in Japanese. This audio is captured by the device, split into 1-second segments, and sent to the server. The server inputs this audio into an AI speech recognition model and converts it into text data: "The next project starts next month." This text data is checked by a filtering mechanism, but since it does not contain any particularly confidential information, it is sent back to the device as is. The device displays this text data as real-time subtitles on the user's screen.

[0860] Example 2: A meeting requiring translation from Japanese to English

[0861] In another scenario, consider a situation where user B says in Japanese, "We have achieved our sales targets." This audio is also captured by the terminal, split, and sent to the server. The server inputs the audio into an AI speech recognition model and generates text data that reads, "We have achieved our sales targets." This text data is then translated into English by a translation tool, resulting in "We have achieved our sales targets." This translation is also checked by a filtering tool and determined not to contain any confidential information. The final text data is then sent to the terminal and displayed as English subtitles.

[0862] Program processing

[0863] 1. Audio data collection: The device captures the user's spoken audio in real time, divides it into appropriate-sized chunks, and sends them to the server.

[0864] 2. Speech Recognition: The server receives the speech data and inputs it into the AI ​​speech recognition model to convert it into text data.

[0865] 3. Translation: If necessary, the converted text data is input into a generated AI translation model and translated into the specified language.

[0866] 4. Filtering: The server detects and masks or replaces confidential company information within the text data.

[0867] 5. Subtitle display: The final text data is sent to the device and displayed on the user's screen in real time.

[0868] In this way, the present invention realizes a system that accurately recognizes speech during meetings, translates it as needed, and displays subtitles in real time while protecting confidential information.

[0869] The following describes the processing flow.

[0870] Step 1:

[0871] Collection of audio data

[0872] User: Speaks during the meeting.

[0873] Terminal: To capture user speech in real time, the system uses the conferencing system's API to obtain the audio stream. The acquired audio data is then divided into segments of a fixed length (e.g., every second).

[0874] Terminal: Stores the divided audio data chunks in a buffer and prepares them for transmission to the server.

[0875] Step 2:

[0876] Sending audio data

[0877] Terminal: Sends audio data chunks stored in the buffer to the server at the appropriate time. Considering network conditions, it is mindful of minimizing latency.

[0878] Step 3:

[0879] Speech recognition

[0880] Server: Receives audio data chunks sent from the terminal.

[0881] Server: Inputs the received audio data chunks into an AI speech recognition model (e.g., Google Speech-to-Text API) to generate the corresponding text data.

[0882] Server: The server processes the text data obtained from the speech recognition model, performing post-processing such as noise reduction and string normalization to improve accuracy.

[0883] Step 4:

[0884] Translation (if necessary)

[0885] Server: Compares the language in which the text data was generated with the display language specified by the user to determine whether translation is necessary.

[0886] Server: When translation is needed, it inputs text data into an AI translation model (e.g., Google Translate API) and translates it into the specified language.

[0887] Server: Review the translation results and make any necessary adjustments.

[0888] Step 5:

[0889] Filtering of confidential company information

[0890] Server: Uses filtering mechanisms to detect confidential company information within the generated text data (including after translation).

[0891] Server: Mask or replace detected confidential information with appropriate tags. This prevents the leakage of sensitive information.

[0892] Server: Generates the final filtered text data.

[0893] Step 6:

[0894] Subtitle data transmission and display

[0895] Server: Prepares to send the final text data to the terminal.

[0896] Server: Sends the final text data to the terminal.

[0897] Terminal: Receives text data sent from the server.

[0898] Terminal: Displays received text data on the user's screen in real time. This allows the user to view what is said during the meeting as subtitles.

[0899] This processing flow enables the system to achieve highly accurate speech recognition and translation, providing users with real-time subtitles while appropriately protecting confidential information.

[0900] (Example 1)

[0901] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0902] In conferencing systems, there is a demand for real-time subtitle display while simultaneously supporting multiple languages ​​and protecting confidential information. However, current technology does not integrate the processes of speech recognition, translation, and filtering, making real-time processing difficult. Furthermore, if the efficient splitting and transmission of audio data are not performed properly, processing delays and reduced accuracy can occur.

[0903] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0904] In this invention, the server includes means for transmitting audio data to the server in real time, speech recognition means for converting the transmitted audio data into text data, translation means for translating the converted text data into different languages, filtering means for detecting, masking, or replacing confidential information from the translated text data, means for transmitting the final text data to a receiving terminal and displaying it in real time, and means for dividing the audio data into appropriate sizes. This enables real-time subtitle display in a conference system while achieving multilingual support and protection of confidential information.

[0905] "Audio data" refers to audio signals or digital data collected during meetings or conversations.

[0906] A "server" is a data processing device or computer system on a network that receives and processes audio data.

[0907] "Text data" refers to character information converted from speech data by speech recognition technology.

[0908] "Speech recognition means" refers to recognition technology or algorithms for converting speech data into text data.

[0909] "Translation methods" refer to techniques or algorithms that convert text data into different languages.

[0910] "Filtering means" refers to techniques or algorithms that detect and mask or replace confidential information from text data.

[0911] A "receiving terminal" is a device that receives the final text data sent from the server and displays it to the user.

[0912] "Means for displaying in real time" refers to a technology or device that displays text data that has undergone conversion and filtering processes without delay.

[0913] "Means for dividing into appropriate sizes" refers to techniques or devices for dividing audio data into appropriate time intervals so that it can be easily processed.

[0914] Modes for carrying out the invention

[0915] This invention relates to a conferencing system that combines speech recognition, translation, confidential information protection, and real-time captioning. This system uses specific hardware and software to collect, transmit, analyze, convert, filter, and display audio data.

[0916] Collection and transmission of audio data

[0917] When a user speaks during a meeting, the device captures their audio in real time. For example, Zoom or Microsoft Teams can be used as the meeting software. The captured audio data is divided into appropriate sizes (e.g., every second) and sent from the device to the server. It is desirable that the audio data be compressed during transmission.

[0918] Speech recognition

[0919] The server receives audio data sent from the terminal in real time. The audio data is input into an AI speech recognition model (e.g., Google Speech-to-Text API) and converted into text data. For example, the statement "The next project starts next month" is converted into the text data "The next project starts next month".

[0920] translation

[0921] The converted text data is then fed into translation models (e.g., DeepL, Google Translate) as needed, and translated into different languages. For example, the Japanese phrase "We have achieved our sales targets" is translated into English as "We have achieved our sales targets."

[0922] filtering

[0923] The server inspects the translated text data and detects sensitive information. Using natural language processing tools (e.g., spaCy, Microsoft Azure Text Analytics), it identifies specific sensitive information and masks or replaces the relevant sections. For example, "We achieved our sales target" might be masked to "Our sales target is..."

[0924] Subtitle display

[0925] The final text data is sent from the server to the terminal. The terminal displays the received text data as real-time subtitles on the user's screen. Subtitle display software such as OBS Studio is used for display.

[0926] Specific example

[0927] Example 1: Japanese meeting

[0928] User A: In a Japanese Zoom meeting, says, "The next project will start next month."

[0929] Terminal: Captures audio in real time, splits it into 1-second segments, and sends them to the server.

[0930] Server: Receives the audio and uses the Google Speech-to-Text API to convert it into text data that says, "The next project starts next month."

[0931] Server: Inspects the text data and, since it does not contain any particularly sensitive information, sends it to the terminal as is.

[0932] Terminal: Displays text data as real-time subtitles.

[0933] Example 2: A meeting requiring translation from Japanese to English

[0934] User B: Says in Japanese, "We have achieved our sales target."

[0935] Terminal: Captures audio and sends it to the server.

[0936] Server: Inputs the audio into an AI speech recognition model and generates text data that says, "Sales target achieved."

[0937] Server: Inputs text data into DeepL and obtains the English translation "We have achieved our sales targets".

[0938] Server: The translation results are checked and determined not to contain any confidential information.

[0939] Device: Display as English subtitles.

[0940] Example of a prompt

[0941] Speech Recognition and Real-Time Display for Japanese Meetings: "Please describe a program that converts Japanese audio during a meeting into text data and displays it in real time."

[0942] Translation and display from Japanese to English: "Please explain the process of translating Japanese audio into English and displaying it as subtitles in real time."

[0943] In this way, the present invention provides a means for efficiently and effectively conducting meetings by integrating speech recognition, translation, confidential information protection, and real-time captioning into a single system within a conference system.

[0944] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0945] Step 1:

[0946] Collection and capture of audio data

[0947] Input: Audio of a user speaking during a meeting.

[0948] Specific action: The user speaks through conferencing software such as Zoom or Teams.

[0949] Output: Audio data captured by the device.

[0950] Processing details: The device captures the user's speech in real time using the microphone. This audio data is saved in digital format and immediately split.

[0951] Step 2:

[0952] Splitting and transmitting audio data

[0953] Input: Captured audio data.

[0954] Specific operation: The device divides the captured audio data into 1-second segments. It also compresses the audio data to reduce unnecessary data.

[0955] Output: Split and compressed audio data.

[0956] Processing details: The terminal divides the captured audio data into appropriate chunks (every second), compresses the data, and sends it to the server. This process improves communication efficiency.

[0957] Step 3:

[0958] Receiving audio data and speech recognition

[0959] Input: Split audio data sent from the device.

[0960] Specific operation: The server receives this data and passes it to an AI speech recognition model (e.g., Google Speech-to-Text API).

[0961] Output: Text data.

[0962] Processing details: The server inputs the received audio data into an AI speech recognition model and converts it into text data in real time. For example, "The next project starts next month" is generated as text data.

[0963] Step 4:

[0964] Text data translation

[0965] Input: Text data generated by speech recognition.

[0966] Specific operation: The server inputs text data into a translation model (e.g., Google Translate, DeepL) according to the requirements.

[0967] Output: Translated text data.

[0968] Processing details: The converted text data is input into a translation model and translated into the specified language. For example, "We have achieved our sales targets" is translated as "We have achieved our sales targets."

[0969] Step 5:

[0970] Filtering of translated text data

[0971] Input: Translated text data.

[0972] Specific operation: The server detects sensitive information within the text data using filtering tools (e.g., spaCy, Microsoft Azure Text Analytics).

[0973] Output: Text data with confidential information masked or replaced.

[0974] Processing details: Inspect the translated text data and mask any confidential information found (e.g., "Sales targets are") or replace it (e.g., "Top Secret Data" → "Important Data").

[0975] Step 6:

[0976] Sending the final text data to the receiving terminal

[0977] Input: Filtered text data.

[0978] Specific operation: The server sends the final text data to the receiving terminal.

[0979] Output: Text data sent to the receiving terminal.

[0980] Processing details: The final text data is sent from the server to the receiving terminal. The sent data is processed at the receiving terminal.

[0981] Step 7:

[0982] Real-time subtitle display

[0983] Input: The final text data sent from the server.

[0984] Specific operation: The receiving terminal displays this text data on its screen in real time. For example, subtitles can be displayed using OBS Studio or similar software.

[0985] Output: Real-time subtitles displayed on the user's screen.

[0986] Processing details: The receiving terminal displays the final text data as real-time subtitles. This makes it easier for users to understand the meeting content.

[0987] (Application Example 1)

[0988] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0989] Communication among passengers in autonomous vehicles presents challenges such as language barriers and the risk of confidential information leakage. Furthermore, while real-time speech recognition, translation, and the protection and display of confidential information are necessary, there is a lack of systems that can efficiently achieve these goals. This challenge is particularly pronounced in multilingual environments and business meetings where the risk of information leakage is high.

[0990] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0991] In this invention, the server includes means for collecting voice data, means for transmitting the collected voice data to the server in real time, and voice recognition means that uses a generative AI model to convert the voice data into text data. This enables support for communication between multiple languages ​​and protection of confidential information.

[0992] A "conference system and in-vehicle communication system" is a system that has functions such as voice data collection, speech recognition, translation, confidential information protection, and real-time display, and operates effectively in multilingual environments and situations with a high risk of information leakage.

[0993] "Means for collecting audio data" refers to devices such as microphones and digital audio recorders used to capture audio in real time from inside a vehicle or conference room.

[0994] "Means of sending to the server in real time" refers to network interfaces and data transfer protocols for sending captured audio data to the server without delay.

[0995] "Speech recognition means" refers to software or hardware used to convert speech data into text data, particularly those that utilize generative AI models.

[0996] "Translation means" refers to software or generative AI models used to translate text data obtained by speech recognition means into different languages.

[0997] "Filtering means" refers to algorithms or software used to detect confidential information from translated text data and to mask or replace that information.

[0998] "Means of sending to a receiving terminal and displaying in real time" refers to programs or hardware that send the final text data to a client device and display it on that device in real time.

[0999] "Means for displaying text data in real time on an in-vehicle display" refers to a device that has an interface function for displaying text data in real time on a screen or monitor installed inside a vehicle.

[1000] A "generative AI model" is an artificial intelligence algorithm trained to perform tasks such as speech recognition, translation, and sensitive information detection.

[1001] This invention is a system for supporting communication within autonomous vehicles and conference systems. The "Conference System and In-Vehicle Communication System" has functions such as voice data collection, speech recognition, translation, confidential information protection, and real-time display. This system operates particularly effectively in multilingual environments and situations with a high risk of information leakage.

[1002] 1. Collection methods

[1003] The server uses microphones or digital audio recorders installed inside the vehicle or conference room as a means of collecting audio data. This allows for real-time capture of user speech.

[1004] 2. Transmission method

[1005] The collected audio data is transmitted to the server in real time, using a network interface and data transfer protocol. The server divides the received audio data into appropriate sizes and proceeds to the next processing step.

[1006] 3. Speech recognition means

[1007] The transmitted audio data is converted into text data by a speech recognition system that uses a generative AI model. Specifically, the Google Cloud Speech-to-Text API is used. This speech recognition system achieves high-precision speech-to-text conversion.

[1008] 4. Translation methods

[1009] The converted text data is translated into different languages ​​as needed using translation tools. This utilizes generative AI models such as the Google Cloud Translate API. This facilitates smooth communication between languages.

[1010] 5. Filtering means

[1011] The translated text data is filtered to detect and mask or replace sensitive information. Regular expressions and custom algorithms are used to ensure that sensitive information is not leaked.

[1012] 6. Display means

[1013] The final text data is sent to the receiving terminal and displayed in real time. Passengers can check the spoken content and translation results in real time using the in-vehicle display. This display is processed using the Google Cloud Speech-to-Text API and the Google Cloud Translate API.

[1014] Examples of specific cases and prompt statements

[1015] For example, if you say "This project is a secret" in Japanese, it will be processed as follows:

[1016] 1. Audio data collection: The in-car microphone captures the phrase "This project is confidential."

[1017] 2. Speech Recognition: The Google Cloud Speech-to-Text API was used to convert the text to "This project is a secret."

[1018] 3. Translation: Translated to "This project is confidential" using the Google Cloud Translate API.

[1019] 4. Filtering: Detect the word "secret" and replace it with "[filtered]".

[1020] 5. Display: The in-vehicle display will show "This project is [filtered]".

[1021] Examples of prompt statements are as follows:

[1022] Please use the Google Cloud Speech-to-Text API to transcribe live meeting audio.

[1023] Please translate the Japanese text into English using the Google Cloud Translate API.

[1024] Detect the words "secret" and "confidential" in the text and replace them with "[filtered]".

[1025] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[1026] Step 1:

[1027] Collection of audio data

[1028] Subject: terminal

[1029] Operation: When a user speaks inside an autonomous vehicle, their voice is captured in real time on the device.

[1030] Input: User's spoken audio

[1031] Data processing / data calculation: Audio data is captured in an appropriate format (e.g., LINEAR16).

[1032] Output: Audio data file (byte data)

[1033] Step 2:

[1034] Sending audio data

[1035] Subject: terminal

[1036] Operation: Audio data captured on the device is sent to the server in real time via the network.

[1037] Input: Audio data file (byte data)

[1038] Data processing / data calculation: Audio data is divided into small chunks to make it easier to transmit over the network.

[1039] Output: Divided audio data chunks

[1040] Step 3:

[1041] Speech recognition

[1042] Subject: Server

[1043] Operation: Audio data sent to the server is converted into text data using a generative AI model.

[1044] Input: Split audio data chunks

[1045] Data processing / data calculation: Convert audio data to text data using the Google Cloud Speech-to-Text API.

[1046] Output: Text data

[1047] Step 4:

[1048] translation

[1049] Subject: Server

[1050] Operation: Text data is translated into different languages ​​as needed.

[1051] Input: Text data

[1052] Data processing / data calculation: Translate text data into a specified language using the Google Cloud Translate API.

[1053] Output: Translated text data

[1054] Step 5:

[1055] filtering

[1056] Subject: Server

[1057] Operation: Sensitive information is detected from translated text data and masked or replaced.

[1058] Input: Translated text data

[1059] Data processing / data calculations: Detect sensitive information using regular expressions and custom algorithms, and perform masking or replacement.

[1060] Output: Filtered text data

[1061] Step 6:

[1062] Sending text data

[1063] Subject: Server

[1064] Operation: The final text data is sent to the receiving terminal.

[1065] Input: Filtered text data

[1066] Data processing / data calculation: Encode data in the appropriate format and prepare it for transmission.

[1067] Output: Text data to be sent

[1068] Step 7:

[1069] Real-time display

[1070] Subject: terminal

[1071] Operation: The final text data is displayed in real time on the in-vehicle display.

[1072] Input: Submitted text data

[1073] Data processing / data calculation: Formatting text data for screen display.

[1074] Output: Text data displayed on the screen

[1075] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[1076] This invention relates to a conferencing system that combines speech recognition, translation, confidential information protection, emotion recognition, and real-time captioning. The system aims to collect audio data, transmit it to a server, perform speech recognition to convert it into text data, translate and filter it as needed, recognize the user's emotions, and finally display it on the terminal in real time.

[1077] System Overview

[1078] This system is configured as follows: First, when a user speaks during a meeting, their voice is captured in real time by the terminal. The captured voice data is appropriately segmented and sent to the server. The server receives the voice data and converts it into text data using an AI speech recognition model. The converted text data is translated into different languages ​​by translation means as needed. Subsequently, sensitive information is detected and masked or replaced in the text data by filtering means. Furthermore, the emotion engine identifies the user's emotional state from the voice data, and this information is added to the text data. The final text data is sent back to the terminal and displayed to the user in real time.

[1079] Specific example

[1080] Example 1: Japanese meeting

[1081] Let's say User A says "The next project starts next month" during a Zoom meeting in Japanese. This audio is captured by the device, split into 1-second segments, and sent to the server. The server inputs this audio into an AI speech recognition model and converts it into text data: "The next project starts next month." This text data is checked by a filtering mechanism, but since it does not contain any particularly confidential information, it is sent back to the device as is. The emotion engine identifies User A's emotions (e.g., joy or anxiety) from the audio data and adds that information to the text data. The device displays this text data as real-time subtitles on the user's screen.

[1082] Example 2: A meeting requiring translation from Japanese to English

[1083] In another scenario, consider a situation where User B says in Japanese, "We have achieved our sales targets." This audio is also captured by the device, split, and sent to the server. The server inputs the audio into an AI speech recognition model and generates text data that reads, "We have achieved our sales targets." This text data is then translated into English by a translation tool, resulting in "We have achieved our sales targets." This translation is also checked by a filtering tool to ensure it does not contain any confidential information. The emotion engine then identifies User B's emotions (e.g., a sense of accomplishment or relief) from the audio data and adds this information to the text data. The final text data is sent to the device and displayed as English subtitles.

[1084] Program processing

[1085] 1. Audio data collection: The device captures the user's spoken audio in real time, divides it into appropriate-sized chunks, and sends them to the server.

[1086] 2. Speech Recognition: The server receives the speech data and inputs it into the AI ​​speech recognition model to convert it into text data.

[1087] 3. Translation: If necessary, the converted text data is input into a generated AI translation model and translated into the specified language.

[1088] 4. Filtering: The server detects and masks or replaces confidential company information within the text data.

[1089] 5. Emotion Recognition: The emotion engine recognizes the user's emotions from the audio data and adds that emotion information to the text data.

[1090] 6. Subtitle Display: The final text data is sent to the device and displayed on the user's screen in real time. This allows the user to see subtitles that include the words spoken during the meeting and the emotions they conveyed.

[1091] In this way, the present invention realizes a system that accurately recognizes speech during meetings, translates it as needed, and displays subtitles in real time, including user sentiment information, while appropriately protecting confidential information.

[1092] The following describes the processing flow.

[1093] Step 1:

[1094] Collection of audio data

[1095] User: Speaks during the meeting.

[1096] Terminal: To capture user speech in real time, the system uses the conferencing system's API to obtain the audio stream. The acquired audio data is then divided into segments of a fixed length (e.g., every second).

[1097] Terminal: Stores the divided audio data chunks in a buffer and prepares them for transmission to the server.

[1098] Step 2:

[1099] Sending audio data

[1100] Terminal: Sends audio data chunks stored in the buffer to the server at the appropriate time. Considering network conditions, it is mindful of minimizing latency.

[1101] Step 3:

[1102] Speech recognition

[1103] Server: Receives audio data chunks sent from the terminal.

[1104] Server: Inputs the received audio data chunks into an AI speech recognition model (e.g., Google Speech-to-Text API) to generate the corresponding text data.

[1105] Server: The server processes the text data obtained from the speech recognition model, performing post-processing such as noise reduction and string normalization to improve accuracy.

[1106] Step 4:

[1107] Translation (if necessary)

[1108] Server: Compares the language in which the text data was generated with the display language specified by the user to determine whether translation is necessary.

[1109] Server: When translation is needed, it inputs text data into an AI translation model (e.g., Google Translate API) and translates it into the specified language.

[1110] Server: Review the translation results and make any necessary adjustments.

[1111] Step 5:

[1112] Filtering of confidential company information

[1113] Server: Uses filtering mechanisms to detect confidential company information within the generated text data (including after translation).

[1114] Server: Mask or replace detected confidential information with appropriate tags. This prevents the leakage of sensitive information.

[1115] Server: Generates the final filtered text data.

[1116] Step 6:

[1117] emotion recognition

[1118] Server: Inputs voice data into the emotion engine to identify the user's emotional state.

[1119] Server: Adds identified sentiment information to the text data. At this time, it clarifies which specific statement the sentiment information corresponds to.

[1120] Step 7:

[1121] Subtitle data transmission and display

[1122] Server: Prepares to send the final text data to the terminal.

[1123] Server: Sends the final text data to the terminal.

[1124] Terminal: Receives text data sent from the server.

[1125] Terminal: Displays received text data on the user's screen in real time. This allows the user to see what is being said in a meeting as subtitles and simultaneously understand the speaker's emotions.

[1126] This processing flow enables the system to achieve highly accurate speech recognition and translation, providing real-time subtitles that include user sentiment information while appropriately protecting confidential data.

[1127] (Example 2)

[1128] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1129] Conventional conferencing systems often handle speech recognition, translation, and confidential information protection separately, lacking a system that integrates and processes all of these in real time. Furthermore, the lack of a function to recognize and display user emotions can lead to a decline in communication quality. To address these problems, the present invention aims to provide a real-time captioning system that combines speech recognition, translation, confidential information protection, and emotion recognition.

[1130] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[1131] In this invention, the server includes speech recognition means, translation means, filtering means, emotion recognition means, and means for transmitting the final text data to a receiving terminal and displaying it in real time. This makes it possible to recognize speech during a meeting in real time, translate it as needed, and display subtitles in real time, including the user's emotion information, while appropriately protecting confidential information.

[1132] "Audio data" refers to information that represents audio signals in digital format, capturing the content of the user's speech.

[1133] "Collection means" refers to a device or software for collecting voice data spoken by a user.

[1134] "Transmission means" refers to a device or software for transmitting collected audio data to a server in real time.

[1135] "Speech recognition means" refers to a device or software for converting speech data into text data.

[1136] "Translation means" refers to a device or software for translating converted text data into a different language.

[1137] "Filtering means" refers to a device or software for detecting, masking, or replacing confidential information from translated text data.

[1138] "Emotion recognition means" refers to a device or software that identifies a user's emotions from audio data and adds that information to text data.

[1139] "Display means" refers to a device or software that transmits the final text data to a receiving terminal and displays it on the user's screen in real time.

[1140] The system of this invention collects audio data spoken during a meeting in real time and performs speech recognition, translation, confidential information protection, sentiment recognition, and real-time subtitle display. This system consists of a server, terminals, and users.

[1141] First, the device captures the user's speech in real time. This capture uses an audio input device (typically a microphone). For example, if the user is using conferencing software such as Zoom or Teams, the audio capture function built into that application is used. The captured audio data is divided into appropriate chunk sizes and sent to the server.

[1142] The server feeds the received audio data into a speech recognition system. This speech recognition system uses a generative artificial intelligence model such as the Google Speech-to-Text API to convert the audio data into highly accurate text data. The converted text data is then sent to a translation system as needed to be translated into the specified language. For example, using the DeepL API, the Japanese text "We have achieved our sales targets" can be translated into the English text "We have achieved our sales targets".

[1143] Next, the server uses filtering mechanisms to detect sensitive information within the converted and translated text data. Using the Data Loss Prevention API, etc., if sensitive information is found, that portion is masked or replaced. For example, if "sales target" is determined to be sensitive information, it will be masked to something like "We achieved our target."

[1144] Furthermore, the server uses emotion recognition to identify the user's emotions from the audio data and adds that information to the text data. Google Cloud's Natural Language API, among others, is used for this emotion recognition. For example, if the user's emotions (such as joy or accomplishment) are detected from the tone and pitch of their voice, that emotion information is added to the text data as a label.

[1145] Once the final text data is generated, the server sends it back to the terminal. The terminal displays the returned text data on the user's screen in real time. This allows the user to see subtitles of what was said during the meeting, including the emotions expressed. For example, it might display something like, "The next project starts next month [excited]."

[1146] Specific example

[1147] If user A says "The next project starts next month" during a Zoom meeting, this audio is captured on the device, split, and sent to the server. The server feeds this audio into the Google Speech-to-Text API to generate text data that reads "The next project starts next month." This data is then checked by the Data Loss Prevention API and determined to be free of confidential information. Furthermore, an emotion recognition engine adds emotion information, such as "joy." This final text data is then sent to the device and displayed in real time.

[1148] In another scenario, User B says in Japanese, "We have achieved our sales targets." This audio is captured on the device, split, and sent to the server. The server uses the Google Speech-to-Text API and the DeepL API to translate this audio into the English text, "We have achieved our sales targets." The Data Loss Prevention API then determines that no sensitive information is included. An emotion recognition engine adds the emotion "sense of accomplishment." The resulting text data is sent to the device and displayed in real time.

[1149] Example of a prompt

[1150] Speech recognition prompt:

[1151] Please convert "The next project will start next month" into text.

[1152] Translation prompt:

[1153] Please translate "We achieved our sales target" into English.

[1154] Emotion recognition prompt:

[1155] Identify the user's emotions from the audio data and add annotations to the results.

[1156] Thus, the system of the present invention improves user communication by recognizing, translating, filtering, and recognizing the emotions of speech during meetings in real time, and displaying it as subtitles.

[1157] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1158] Specific flow of the system program's processing

[1159] Step 1:

[1160] Collection of audio data

[1161] The device uses an audio input device (microphone) to capture the user's spoken audio in real time. For example, if a user says "The next project starts next month" in a Zoom meeting, this audio will be captured.

[1162] Input: User's spoken audio.

[1163] Specific operation: The captured audio data is divided into chunks of one second each. The terminal then sends this chunked audio data to the next processing step.

[1164] Step 2:

[1165] Sending audio data

[1166] The terminal sends chunked audio data to the server in real time.

[1167] Input: Audio data chunked every second.

[1168] Specific operation: The device uploads this data to the server via the internet. For example, real-time transfer using WebSocket is a possible approach.

[1169] Step 3:

[1170] Speech recognition

[1171] The server inputs the received audio data into an AI speech recognition model and converts it into text data. APIs such as Google Speech-to-Text are used for this purpose.

[1172] Input: Audio data sent from the device.

[1173] Output: Converted text data. (Example: "The next project starts next month.")

[1174] Specific operation: The server processes the audio chunks sequentially to generate high-precision text data. This data is then sent to subsequent processing steps.

[1175] Step 4:

[1176] translation

[1177] The server feeds the generated text data into the translation system as needed, and translates it into the desired language. The DeepL API is used for translation.

[1178] Input: Converted text data. (Example: "The next project starts next month.")

[1179] Output: Translated text data. (Example: "The next project will start from next month")

[1180] Specific operation: The server sends text data based on the configured target language and receives the translation result. This data is then sent to the next filtering step.

[1181] Step 5:

[1182] filtering

[1183] The server filters the translated text data and takes measures to detect sensitive information. Data Loss Prevention APIs, among others, are used.

[1184] Input: Translated text data. (Example: "The next project will start from next month")

[1185] Output: Filtered text data. (If there is confidential information, e.g., "The next project will start from next month")

[1186] Specific operation: The server scans the text data and detects keywords related to sensitive information. Those parts are then replaced or masked.

[1187] Step 6:

[1188] emotion recognition

[1189] The server uses emotion recognition to identify the user's emotions from the audio data and adds that information to the text data. Google Cloud's Natural Language API is used.

[1190] Input: User's voice data and filtered text data. (Example: "The next project will from next month")

[1191] Output: Text data with emotion labels. (Example: "The next project will from next month [joy]")

[1192] Specific operation: Emotional features such as tone and pitch are extracted from audio data, and the detected emotions (joy, anxiety, etc.) are added as labels to the text data.

[1193] Step 7:

[1194] Display subtitles

[1195] The server sends the final text data to the terminal and displays it on the user's screen in real time.

[1196] Input: Text data with emotion labels attached. (Example: "The next project will from next month [joy]")

[1197] Output: Subtitles displayed on the user's device. (Example: "The next project will be from next month [joy]")

[1198] Specific operation: The device displays this data in the caption box of the conferencing software. This allows users to see the content and sentiment information of what is being said in real time.

[1199] This series of processing steps enables a system that recognizes, translates, filters, and recognizes the sentiment of audio data during a meeting in real time, and displays it to the user immediately.

[1200] (Application Example 2)

[1201] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1202] Traditional meeting and customer service systems had limited voice recognition and translation capabilities, making them inadequate for real-time communication, multilingual support, and the protection of confidential information. Furthermore, the lack of means to recognize and provide real-time information about the emotions of customers and meeting participants resulted in a decline in communication quality. Additionally, the reliability of translation results and the protection of confidential information were insufficient, necessitating appropriate solutions.

[1203] In Application Example 2, the specific processing performed by the specific processing unit 290 of the data processing device 12 is realized by the following means. In this invention, the server includes means for collecting voice data, means for transmitting the collected voice data to the server in real time, voice recognition means for converting the transmitted voice data into text data, translation means for translating the converted text data into different languages, filtering means for detecting, masking, or replacing confidential information from the translated text data, emotion recognition means for identifying emotional states from the voice data and adding that information to the text data, and means for transmitting the final text data to a receiving terminal and displaying it in real time. This enables voice recognition, translation, protection of confidential information, and real-time display including emotion recognition.

[1204] "Audio data" refers to a data format in which sound is recorded electronically and can be analyzed and converted through computer processing.

[1205] "Means of collection" refers to devices or systems that capture audio data in real time and store it in digital format.

[1206] "Means of sending to the server in real time" refers to communication methods for transferring collected audio data to the server without delay.

[1207] "Speech recognition means" refers to a software or hardware configuration for analyzing speech data and converting it into text data.

[1208] "Translation means" refers to software or services for converting text data into different languages.

[1209] A "filtering method" is an algorithm that automatically detects confidential information from text data and masks or replaces that portion.

[1210] "Emotion recognition means" refers to software or models that detect emotions from audio or text data and add that information.

[1211] "Final text data" refers to the character data after processing is complete, including all processing such as speech recognition, translation, filtering, and sentiment recognition.

[1212] A "receiving terminal" is a device used to display the final text data, and includes smartphones, smart glasses, and other similar devices.

[1213] The embodiments for carrying out the present invention will be described in detail below.

[1214] This system processes voice data collection, speech recognition, translation, filtering, sentiment recognition, and real-time display as a series of processes. The voice spoken by the user is first captured in real time by a dedicated device (such as smart glasses or a smartphone). Suitable hardware includes a microphone and a high-speed central processing unit (CPU).

[1215] After the collection of audio data is complete, the collected data is transmitted to the server in real time. High-speed communication lines such as Wi-Fi or 4G / 5G are used for this transmission. The server first appropriately divides (chunks) the received audio data into digital format, and then divides it into appropriate sizes according to the amount of data.

[1216] The segmented audio data is then converted into text data using speech recognition software such as the Google Cloud Speech-to-Text API or IBM Watson Speech to Text. The converted text data is then translated into the specified language using the Google Cloud Translation API, if necessary.

[1217] The translated text data is further filtered to automatically detect and mask or replace personal or confidential information. This minimizes the risk of information leakage.

[1218] Subsequently, IBM Watson Tone Analyzer or a similar emotion recognition engine is used to identify the user's emotional state from the text data. Emotional information is added to the text data, and the final text data including this information is generated.

[1219] This final text data is sent back to the device and displayed to the user in real time. By displaying it on a screen such as smart glasses or a smartphone, the user can see not only what they have said, but also the emotions and important information contained within that statement in real time.

[1220] Specific example:

[1221] The following prompt messages are a concrete example of how this system can be applied to a customer service system in a physical store.

[1222] "Development of a system that recognizes emotions from customer statements, performs necessary translations, and displays them in real time on smart glasses."

[1223] Follow these steps to develop a system that recognizes emotions from customer speech, performs necessary translations, and displays them in real time on smart glasses.

[1224] 1. Capture customer voices in real time and convert them to text.

[1225] 2. If necessary, translate the text into the specified language.

[1226] 3. Detect and mask confidential information from the text.

[1227] 4. Recognize customer emotions from voice and add that information to text.

[1228] 5. Display the final text data on the smart glasses' display in real time.

[1229] This system facilitates multilingual support and enables more effective customer service by recognizing customer emotions. Furthermore, it provides a user-friendly environment while appropriately protecting confidential information.

[1230] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1231] Step 1:

[1232] The device captures the user's spoken voice in real time and converts it into a digital format. Specifically, the device's microphone collects the voice and saves it as an audio file. The input to this process is the user's voice data, and the output is a digital audio file.

[1233] Step 2:

[1234] The terminal transmits captured audio data to the server in real time. Specifically, the terminal uses a high-speed communication line such as Wi-Fi or 4G / 5G to divide the audio data into data packets and send them to the server. The input to this process is an audio file, and the output is the audio data sent to the server.

[1235] Step 3:

[1236] The server divides the received audio data into appropriate-sized chunks. Specifically, the server divides the audio file at regular time intervals (e.g., 1 second) and saves each chunk separately. The input to this process is the transmitted audio data, and the output is the divided audio chunks.

[1237] Step 4:

[1238] The server inputs the divided audio chunks into a speech recognition system and converts them into text data. Specifically, the server calls the Google Cloud Speech-to-Text API to convert the audio data into text data. The input to this process is audio chunks, and the output is text data.

[1239] Step 5:

[1240] The server translates the converted text data into a different language. Specifically, the server uses the Google Cloud Translation API to translate the text data into the specified target language. The input to this process is the text data, and the output is the translated text data.

[1241] Step 6:

[1242] The server detects sensitive information from translated text data and masks or replaces it. Specifically, the server uses regular expressions and machine learning models to identify personal and sensitive information and replaces the detected information with masking characters such as "". The input to this process is translated text data, and the output is filtered text data.

[1243] Step 7:

[1244] The server inputs filtered text data into an emotion recognition system, identifies the user's emotional state, and adds that information to the text data. Specifically, the server uses IBM Watson Tone Analyzer to perform emotion analysis and adds the results to the original text data. The input to this process is filtered text data, and the output is text data with added emotion information.

[1245] Step 8:

[1246] The server sends the final text data to the receiving terminal and displays it in real time. Specifically, the server sends the generated text data to the terminal as real-time subtitles, which the terminal displays on its screen. The input to this process is text data with added emotional information, and the output is real-time subtitles displayed on the user's screen.

[1247] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1248] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1249] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[1250] [Fourth Embodiment]

[1251] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[1252] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1253] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1254] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[1255] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[1256] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[1257] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[1258] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[1259] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[1260] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1261] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1262] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[1263] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1264] This invention relates to a system that combines speech recognition, translation, confidential information protection, and real-time captioning in a conference system. The system aims to collect audio data, transmit it to a server, perform speech recognition to convert it into text data, translate and filter it as needed, and finally display it on a terminal in real time.

[1265] System Overview

[1266] This system is configured as follows: First, when a user speaks during a meeting, their voice is captured in real time by the terminal. The captured voice data is appropriately segmented and sent to the server. The server receives the voice data and converts it into text data using an AI speech recognition model. The converted text data is translated into different languages ​​by translation means as needed. Subsequently, sensitive information is detected and masked or replaced in the text data by filtering means. The final text data is sent back to the terminal and displayed to the user in real time.

[1267] Specific example

[1268] Example 1: Japanese meeting

[1269] Let's say User A says "The next project starts next month" during a Zoom meeting in Japanese. This audio is captured by the device, split into 1-second segments, and sent to the server. The server inputs this audio into an AI speech recognition model and converts it into text data: "The next project starts next month." This text data is checked by a filtering mechanism, but since it does not contain any particularly confidential information, it is sent back to the device as is. The device displays this text data as real-time subtitles on the user's screen.

[1270] Example 2: A meeting requiring translation from Japanese to English

[1271] In another scenario, consider a situation where user B says in Japanese, "We have achieved our sales targets." This audio is also captured by the terminal, split, and sent to the server. The server inputs the audio into an AI speech recognition model and generates text data that reads, "We have achieved our sales targets." This text data is then translated into English by a translation tool, resulting in "We have achieved our sales targets." This translation is also checked by a filtering tool and determined not to contain any confidential information. The final text data is then sent to the terminal and displayed as English subtitles.

[1272] Program processing

[1273] 1. Audio data collection: The device captures the user's spoken audio in real time, divides it into appropriate-sized chunks, and sends them to the server.

[1274] 2. Speech Recognition: The server receives the speech data and inputs it into the AI ​​speech recognition model to convert it into text data.

[1275] 3. Translation: If necessary, the converted text data is input into a generated AI translation model and translated into the specified language.

[1276] 4. Filtering: The server detects and masks or replaces confidential company information within the text data.

[1277] 5. Subtitle display: The final text data is sent to the device and displayed on the user's screen in real time.

[1278] In this way, the present invention realizes a system that accurately recognizes speech during meetings, translates it as needed, and displays subtitles in real time while protecting confidential information.

[1279] The following describes the processing flow.

[1280] Step 1:

[1281] Collection of audio data

[1282] User: Speaks during the meeting.

[1283] Terminal: To capture user speech in real time, the system uses the conferencing system's API to obtain the audio stream. The acquired audio data is then divided into segments of a fixed length (e.g., every second).

[1284] Terminal: Stores the divided audio data chunks in a buffer and prepares them for transmission to the server.

[1285] Step 2:

[1286] Sending audio data

[1287] Terminal: Sends audio data chunks stored in the buffer to the server at the appropriate time. Considering network conditions, it is mindful of minimizing latency.

[1288] Step 3:

[1289] Speech recognition

[1290] Server: Receives audio data chunks sent from the terminal.

[1291] Server: Inputs the received audio data chunks into an AI speech recognition model (e.g., Google Speech-to-Text API) to generate the corresponding text data.

[1292] Server: The server processes the text data obtained from the speech recognition model, performing post-processing such as noise reduction and string normalization to improve accuracy.

[1293] Step 4:

[1294] Translation (if necessary)

[1295] Server: Compares the language in which the text data was generated with the display language specified by the user to determine whether translation is necessary.

[1296] Server: When translation is needed, it inputs text data into an AI translation model (e.g., Google Translate API) and translates it into the specified language.

[1297] Server: Review the translation results and make any necessary adjustments.

[1298] Step 5:

[1299] Filtering of confidential company information

[1300] Server: Uses filtering mechanisms to detect confidential company information within the generated text data (including after translation).

[1301] Server: Mask or replace detected confidential information with appropriate tags. This prevents the leakage of sensitive information.

[1302] Server: Generates the final filtered text data.

[1303] Step 6:

[1304] Subtitle data transmission and display

[1305] Server: Prepares to send the final text data to the terminal.

[1306] Server: Sends the final text data to the terminal.

[1307] Terminal: Receives text data sent from the server.

[1308] Terminal: Displays received text data on the user's screen in real time. This allows the user to view what is said during the meeting as subtitles.

[1309] This processing flow enables the system to achieve highly accurate speech recognition and translation, providing users with real-time subtitles while appropriately protecting confidential information.

[1310] (Example 1)

[1311] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1312] In conferencing systems, there is a demand for real-time subtitle display while simultaneously supporting multiple languages ​​and protecting confidential information. However, current technology does not integrate the processes of speech recognition, translation, and filtering, making real-time processing difficult. Furthermore, if the efficient splitting and transmission of audio data are not performed properly, processing delays and reduced accuracy can occur.

[1313] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[1314] In this invention, the server includes means for transmitting audio data to the server in real time, speech recognition means for converting the transmitted audio data into text data, translation means for translating the converted text data into different languages, filtering means for detecting, masking, or replacing confidential information from the translated text data, means for transmitting the final text data to a receiving terminal and displaying it in real time, and means for dividing the audio data into appropriate sizes. This enables real-time subtitle display in a conference system while achieving multilingual support and protection of confidential information.

[1315] "Audio data" refers to audio signals or digital data collected during meetings or conversations.

[1316] A "server" is a data processing device or computer system on a network that receives and processes audio data.

[1317] "Text data" refers to character information converted from speech data by speech recognition technology.

[1318] "Speech recognition means" refers to recognition technology or algorithms for converting speech data into text data.

[1319] "Translation methods" refer to techniques or algorithms that convert text data into different languages.

[1320] "Filtering means" refers to techniques or algorithms that detect and mask or replace confidential information from text data.

[1321] A "receiving terminal" is a device that receives the final text data sent from the server and displays it to the user.

[1322] "Means for displaying in real time" refers to a technology or device that displays text data that has undergone conversion and filtering processes without delay.

[1323] "Means for dividing into appropriate sizes" refers to techniques or devices for dividing audio data into appropriate time intervals so that it can be easily processed.

[1324] Modes for carrying out the invention

[1325] This invention relates to a conferencing system that combines speech recognition, translation, confidential information protection, and real-time captioning. This system uses specific hardware and software to collect, transmit, analyze, convert, filter, and display audio data.

[1326] Collection and transmission of audio data

[1327] When a user speaks during a meeting, the device captures their audio in real time. For example, Zoom or Microsoft Teams can be used as the meeting software. The captured audio data is divided into appropriate sizes (e.g., every second) and sent from the device to the server. It is desirable that the audio data be compressed during transmission.

[1328] Speech recognition

[1329] The server receives audio data sent from the terminal in real time. The audio data is input into an AI speech recognition model (e.g., Google Speech-to-Text API) and converted into text data. For example, the statement "The next project starts next month" is converted into the text data "The next project starts next month".

[1330] translation

[1331] The converted text data is then fed into translation models (e.g., DeepL, Google Translate) as needed, and translated into different languages. For example, the Japanese phrase "We have achieved our sales targets" is translated into English as "We have achieved our sales targets."

[1332] filtering

[1333] The server inspects the translated text data and detects sensitive information. Using natural language processing tools (e.g., spaCy, Microsoft Azure Text Analytics), it identifies specific sensitive information and masks or replaces the relevant sections. For example, "We achieved our sales target" might be masked to "Our sales target is..."

[1334] Subtitle display

[1335] The final text data is sent from the server to the terminal. The terminal displays the received text data as real-time subtitles on the user's screen. Subtitle display software such as OBS Studio is used for display.

[1336] Specific example

[1337] Example 1: Japanese meeting

[1338] User A: In a Japanese Zoom meeting, says, "The next project will start next month."

[1339] Terminal: Captures audio in real time, splits it into 1-second segments, and sends them to the server.

[1340] Server: Receives the audio and uses the Google Speech-to-Text API to convert it into text data that says, "The next project starts next month."

[1341] Server: Inspects the text data and, since it does not contain any particularly sensitive information, sends it to the terminal as is.

[1342] Terminal: Displays text data as real-time subtitles.

[1343] Example 2: A meeting requiring translation from Japanese to English

[1344] User B: Says in Japanese, "We have achieved our sales target."

[1345] Terminal: Captures audio and sends it to the server.

[1346] Server: Inputs the audio into an AI speech recognition model and generates text data that says, "Sales target achieved."

[1347] Server: Inputs text data into DeepL and obtains the English translation "We have achieved our sales targets".

[1348] Server: The translation results are checked and determined not to contain any confidential information.

[1349] Device: Display as English subtitles.

[1350] Example of a prompt

[1351] Speech Recognition and Real-Time Display for Japanese Meetings: "Please describe a program that converts Japanese audio during a meeting into text data and displays it in real time."

[1352] Translation and display from Japanese to English: "Please explain the process of translating Japanese audio into English and displaying it as subtitles in real time."

[1353] In this way, the present invention provides a means for efficiently and effectively conducting meetings by integrating speech recognition, translation, confidential information protection, and real-time captioning into a single system within a conference system.

[1354] The flow of the specific processing in Example 1 will be explained using Figure 11.

[1355] Step 1:

[1356] Collection and capture of audio data

[1357] Input: Audio of a user speaking during a meeting.

[1358] Specific action: The user speaks through conferencing software such as Zoom or Teams.

[1359] Output: Audio data captured by the device.

[1360] Processing details: The device captures the user's speech in real time using the microphone. This audio data is saved in digital format and immediately split.

[1361] Step 2:

[1362] Splitting and transmitting audio data

[1363] Input: Captured audio data.

[1364] Specific operation: The device divides the captured audio data into 1-second segments. It also compresses the audio data to reduce unnecessary data.

[1365] Output: Split and compressed audio data.

[1366] Processing details: The terminal divides the captured audio data into appropriate chunks (every second), compresses the data, and sends it to the server. This process improves communication efficiency.

[1367] Step 3:

[1368] Receiving audio data and speech recognition

[1369] Input: Split audio data sent from the device.

[1370] Specific operation: The server receives this data and passes it to an AI speech recognition model (e.g., Google Speech-to-Text API).

[1371] Output: Text data.

[1372] Processing details: The server inputs the received audio data into an AI speech recognition model and converts it into text data in real time. For example, "The next project starts next month" is generated as text data.

[1373] Step 4:

[1374] Text data translation

[1375] Input: Text data generated by speech recognition.

[1376] Specific operation: The server inputs text data into a translation model (e.g., Google Translate, DeepL) according to the requirements.

[1377] Output: Translated text data.

[1378] Processing details: The converted text data is input into a translation model and translated into the specified language. For example, "We have achieved our sales targets" is translated as "We have achieved our sales targets."

[1379] Step 5:

[1380] Filtering of translated text data

[1381] Input: Translated text data.

[1382] Specific operation: The server detects sensitive information within the text data using filtering tools (e.g., spaCy, Microsoft Azure Text Analytics).

[1383] Output: Text data with confidential information masked or replaced.

[1384] Processing details: Inspect the translated text data and mask any confidential information found (e.g., "Sales targets are") or replace it (e.g., "Top Secret Data" → "Important Data").

[1385] Step 6:

[1386] Sending the final text data to the receiving terminal

[1387] Input: Filtered text data.

[1388] Specific operation: The server sends the final text data to the receiving terminal.

[1389] Output: Text data sent to the receiving terminal.

[1390] Processing details: The final text data is sent from the server to the receiving terminal. The sent data is processed at the receiving terminal.

[1391] Step 7:

[1392] Real-time subtitle display

[1393] Input: The final text data sent from the server.

[1394] Specific operation: The receiving terminal displays this text data on its screen in real time. For example, subtitles can be displayed using OBS Studio or similar software.

[1395] Output: Real-time subtitles displayed on the user's screen.

[1396] Processing details: The receiving terminal displays the final text data as real-time subtitles. This makes it easier for users to understand the meeting content.

[1397] (Application Example 1)

[1398] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1399] Communication among passengers in autonomous vehicles presents challenges such as language barriers and the risk of confidential information leakage. Furthermore, while real-time speech recognition, translation, and the protection and display of confidential information are necessary, there is a lack of systems that can efficiently achieve these goals. This challenge is particularly pronounced in multilingual environments and business meetings where the risk of information leakage is high.

[1400] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[1401] In this invention, the server includes means for collecting voice data, means for transmitting the collected voice data to the server in real time, and voice recognition means that uses a generative AI model to convert the voice data into text data. This enables support for communication between multiple languages ​​and protection of confidential information.

[1402] A "conference system and in-vehicle communication system" is a system that has functions such as voice data collection, speech recognition, translation, confidential information protection, and real-time display, and operates effectively in multilingual environments and situations with a high risk of information leakage.

[1403] "Means for collecting audio data" refers to devices such as microphones and digital audio recorders used to capture audio in real time from inside a vehicle or conference room.

[1404] "Means of sending to the server in real time" refers to network interfaces and data transfer protocols for sending captured audio data to the server without delay.

[1405] "Speech recognition means" refers to software or hardware used to convert speech data into text data, particularly those that utilize generative AI models.

[1406] "Translation means" refers to software or generative AI models used to translate text data obtained by speech recognition means into different languages.

[1407] "Filtering means" refers to algorithms or software used to detect confidential information from translated text data and to mask or replace that information.

[1408] "Means of sending to a receiving terminal and displaying in real time" refers to programs or hardware that send the final text data to a client device and display it on that device in real time.

[1409] "Means for displaying text data in real time on an in-vehicle display" refers to a device that has an interface function for displaying text data in real time on a screen or monitor installed inside a vehicle.

[1410] A "generative AI model" is an artificial intelligence algorithm trained to perform tasks such as speech recognition, translation, and sensitive information detection.

[1411] This invention is a system for supporting communication within autonomous vehicles and conference systems. The "Conference System and In-Vehicle Communication System" has functions such as voice data collection, speech recognition, translation, confidential information protection, and real-time display. This system operates particularly effectively in multilingual environments and situations with a high risk of information leakage.

[1412] 1. Collection methods

[1413] The server uses microphones or digital audio recorders installed inside the vehicle or conference room as a means of collecting audio data. This allows for real-time capture of user speech.

[1414] 2. Transmission method

[1415] The collected audio data is transmitted to the server in real time, using a network interface and data transfer protocol. The server divides the received audio data into appropriate sizes and proceeds to the next processing step.

[1416] 3. Speech recognition means

[1417] The transmitted audio data is converted into text data by a speech recognition system that uses a generative AI model. Specifically, the Google Cloud Speech-to-Text API is used. This speech recognition system achieves high-precision speech-to-text conversion.

[1418] 4. Translation methods

[1419] The converted text data is translated into different languages ​​as needed using translation tools. This utilizes generative AI models such as the Google Cloud Translate API. This facilitates smooth communication between languages.

[1420] 5. Filtering means

[1421] The translated text data is filtered to detect and mask or replace sensitive information. Regular expressions and custom algorithms are used to ensure that sensitive information is not leaked.

[1422] 6. Display means

[1423] The final text data is sent to the receiving terminal and displayed in real time. Passengers can check the spoken content and translation results in real time using the in-vehicle display. This display is processed using the Google Cloud Speech-to-Text API and the Google Cloud Translate API.

[1424] Examples of specific cases and prompt statements

[1425] For example, if you say "This project is a secret" in Japanese, it will be processed as follows:

[1426] 1. Audio data collection: The in-car microphone captures the phrase "This project is confidential."

[1427] 2. Speech Recognition: The Google Cloud Speech-to-Text API was used to convert the text to "This project is a secret."

[1428] 3. Translation: Translated to "This project is confidential" using the Google Cloud Translate API.

[1429] 4. Filtering: Detect the word "secret" and replace it with "[filtered]".

[1430] 5. Display: The in-vehicle display will show "This project is [filtered]".

[1431] Examples of prompt statements are as follows:

[1432] Please use the Google Cloud Speech-to-Text API to transcribe live meeting audio.

[1433] Please translate the Japanese text into English using the Google Cloud Translate API.

[1434] Detect the words "secret" and "confidential" in the text and replace them with "[filtered]".

[1435] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[1436] Step 1:

[1437] Collection of audio data

[1438] Subject: terminal

[1439] Operation: When a user speaks inside an autonomous vehicle, their voice is captured in real time on the device.

[1440] Input: User's spoken audio

[1441] Data processing / data calculation: Audio data is captured in an appropriate format (e.g., LINEAR16).

[1442] Output: Audio data file (byte data)

[1443] Step 2:

[1444] Sending audio data

[1445] Subject: terminal

[1446] Operation: Audio data captured on the device is sent to the server in real time via the network.

[1447] Input: Audio data file (byte data)

[1448] Data processing / data calculation: Audio data is divided into small chunks to make it easier to transmit over the network.

[1449] Output: Divided audio data chunks

[1450] Step 3:

[1451] Speech recognition

[1452] Subject: Server

[1453] Operation: Audio data sent to the server is converted into text data using a generative AI model.

[1454] Input: Split audio data chunks

[1455] Data processing / data calculation: Convert audio data to text data using the Google Cloud Speech-to-Text API.

[1456] Output: Text data

[1457] Step 4:

[1458] translation

[1459] Subject: Server

[1460] Operation: Text data is translated into different languages ​​as needed.

[1461] Input: Text data

[1462] Data processing / data calculation: Translate text data into a specified language using the Google Cloud Translate API.

[1463] Output: Translated text data

[1464] Step 5:

[1465] filtering

[1466] Subject: Server

[1467] Operation: Sensitive information is detected from translated text data and masked or replaced.

[1468] Input: Translated text data

[1469] Data processing / data calculations: Detect sensitive information using regular expressions and custom algorithms, and perform masking or replacement.

[1470] Output: Filtered text data

[1471] Step 6:

[1472] Sending text data

[1473] Subject: Server

[1474] Operation: The final text data is sent to the receiving terminal.

[1475] Input: Filtered text data

[1476] Data processing / data calculation: Encode data in the appropriate format and prepare it for transmission.

[1477] Output: Text data to be sent

[1478] Step 7:

[1479] Real-time display

[1480] Subject: terminal

[1481] Operation: The final text data is displayed in real time on the in-vehicle display.

[1482] Input: Submitted text data

[1483] Data processing / data calculation: Formatting text data for screen display.

[1484] Output: Text data displayed on the screen

[1485] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[1486] This invention relates to a conferencing system that combines speech recognition, translation, confidential information protection, emotion recognition, and real-time captioning. The system aims to collect audio data, transmit it to a server, perform speech recognition to convert it into text data, translate and filter it as needed, recognize the user's emotions, and finally display it on the terminal in real time.

[1487] System Overview

[1488] This system is configured as follows: First, when a user speaks during a meeting, their voice is captured in real time by the terminal. The captured voice data is appropriately segmented and sent to the server. The server receives the voice data and converts it into text data using an AI speech recognition model. The converted text data is translated into different languages ​​by translation means as needed. Subsequently, sensitive information is detected and masked or replaced in the text data by filtering means. Furthermore, the emotion engine identifies the user's emotional state from the voice data, and this information is added to the text data. The final text data is sent back to the terminal and displayed to the user in real time.

[1489] Specific example

[1490] Example 1: Japanese meeting

[1491] Let's say User A says "The next project starts next month" during a Zoom meeting in Japanese. This audio is captured by the device, split into 1-second segments, and sent to the server. The server inputs this audio into an AI speech recognition model and converts it into text data: "The next project starts next month." This text data is checked by a filtering mechanism, but since it does not contain any particularly confidential information, it is sent back to the device as is. The emotion engine identifies User A's emotions (e.g., joy or anxiety) from the audio data and adds that information to the text data. The device displays this text data as real-time subtitles on the user's screen.

[1492] Example 2: A meeting requiring translation from Japanese to English

[1493] In another scenario, consider a situation where User B says in Japanese, "We have achieved our sales targets." This audio is also captured by the device, split, and sent to the server. The server inputs the audio into an AI speech recognition model and generates text data that reads, "We have achieved our sales targets." This text data is then translated into English by a translation tool, resulting in "We have achieved our sales targets." This translation is also checked by a filtering tool to ensure it does not contain any confidential information. The emotion engine then identifies User B's emotions (e.g., a sense of accomplishment or relief) from the audio data and adds this information to the text data. The final text data is sent to the device and displayed as English subtitles.

[1494] Program processing

[1495] 1. Audio data collection: The device captures the user's spoken audio in real time, divides it into appropriate-sized chunks, and sends them to the server.

[1496] 2. Speech Recognition: The server receives the speech data and inputs it into the AI ​​speech recognition model to convert it into text data.

[1497] 3. Translation: If necessary, the converted text data is input into a generated AI translation model and translated into the specified language.

[1498] 4. Filtering: The server detects and masks or replaces confidential company information within the text data.

[1499] 5. Emotion Recognition: The emotion engine recognizes the user's emotions from the audio data and adds that emotion information to the text data.

[1500] 6. Subtitle Display: The final text data is sent to the device and displayed on the user's screen in real time. This allows the user to see subtitles of what was said during the meeting, including the emotions conveyed.

[1501] In this way, the present invention realizes a system that accurately recognizes speech during meetings, translates it as needed, and displays subtitles in real time, including user sentiment information, while appropriately protecting confidential information.

[1502] The following describes the processing flow.

[1503] Step 1:

[1504] Collection of audio data

[1505] User: Speaks during the meeting.

[1506] Terminal: To capture user speech in real time, the system uses the conferencing system's API to obtain the audio stream. The acquired audio data is then divided into segments of a fixed length (e.g., every second).

[1507] Terminal: Stores the divided audio data chunks in a buffer and prepares them for transmission to the server.

[1508] Step 2:

[1509] Sending audio data

[1510] Terminal: Sends audio data chunks stored in the buffer to the server at the appropriate time. Considering network conditions, it is mindful of minimizing latency.

[1511] Step 3:

[1512] Speech recognition

[1513] Server: Receives audio data chunks sent from the terminal.

[1514] Server: Inputs the received audio data chunks into an AI speech recognition model (e.g., Google Speech-to-Text API) to generate the corresponding text data.

[1515] Server: The server processes the text data obtained from the speech recognition model, performing post-processing such as noise reduction and string normalization to improve accuracy.

[1516] Step 4:

[1517] Translation (if necessary)

[1518] Server: Compares the language in which the text data was generated with the display language specified by the user to determine whether translation is necessary.

[1519] Server: When translation is needed, it inputs text data into an AI translation model (e.g., Google Translate API) and translates it into the specified language.

[1520] Server: Review the translation results and make any necessary adjustments.

[1521] Step 5:

[1522] Filtering of confidential company information

[1523] Server: Uses filtering mechanisms to detect confidential company information within the generated text data (including after translation).

[1524] Server: Mask or replace detected confidential information with appropriate tags. This prevents the leakage of sensitive information.

[1525] Server: Generates the final filtered text data.

[1526] Step 6:

[1527] emotion recognition

[1528] Server: Inputs voice data into the emotion engine to identify the user's emotional state.

[1529] Server: Adds identified sentiment information to the text data. At this time, it clarifies which specific statement the sentiment information corresponds to.

[1530] Step 7:

[1531] Subtitle data transmission and display

[1532] Server: Prepares to send the final text data to the terminal.

[1533] Server: Sends the final text data to the terminal.

[1534] Terminal: Receives text data sent from the server.

[1535] Terminal: Displays received text data on the user's screen in real time. This allows the user to see what is being said in a meeting as subtitles and simultaneously understand the speaker's emotions.

[1536] This processing flow enables the system to achieve highly accurate speech recognition and translation, providing real-time subtitles that include user sentiment information while appropriately protecting confidential data.

[1537] (Example 2)

[1538] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1539] Conventional conferencing systems often handle speech recognition, translation, and confidential information protection separately, lacking a system that integrates and processes all of these in real time. Furthermore, the lack of a function to recognize and display user emotions can lead to a decline in communication quality. To address these problems, the present invention aims to provide a real-time captioning system that combines speech recognition, translation, confidential information protection, and emotion recognition.

[1540] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[1541] In this invention, the server includes speech recognition means, translation means, filtering means, emotion recognition means, and means for transmitting the final text data to a receiving terminal and displaying it in real time. This makes it possible to recognize speech during a meeting in real time, translate it as needed, and display subtitles in real time, including the user's emotion information, while appropriately protecting confidential information.

[1542] "Audio data" refers to information that represents audio signals in digital format, capturing the content of the user's speech.

[1543] "Collection means" refers to a device or software for collecting voice data spoken by a user.

[1544] "Transmission means" refers to a device or software for transmitting collected audio data to a server in real time.

[1545] "Speech recognition means" refers to a device or software for converting speech data into text data.

[1546] "Translation means" refers to a device or software for translating converted text data into a different language.

[1547] "Filtering means" refers to a device or software for detecting, masking, or replacing confidential information from translated text data.

[1548] "Emotion recognition means" refers to a device or software that identifies a user's emotions from audio data and adds that information to text data.

[1549] "Display means" refers to a device or software that transmits the final text data to a receiving terminal and displays it on the user's screen in real time.

[1550] The system of this invention collects audio data spoken during a meeting in real time and performs speech recognition, translation, confidential information protection, sentiment recognition, and real-time subtitle display. This system consists of a server, terminals, and users.

[1551] First, the device captures the user's speech in real time. This capture uses an audio input device (typically a microphone). For example, if the user is using conferencing software such as Zoom or Teams, the audio capture function built into that application is used. The captured audio data is divided into appropriate chunk sizes and sent to the server.

[1552] The server feeds the received audio data into a speech recognition system. This speech recognition system uses a generative artificial intelligence model such as the Google Speech-to-Text API to convert the audio data into highly accurate text data. The converted text data is then sent to a translation system as needed to be translated into the specified language. For example, using the DeepL API, the Japanese text "We have achieved our sales targets" can be translated into the English text "We have achieved our sales targets".

[1553] Next, the server uses filtering mechanisms to detect sensitive information within the converted and translated text data. Using the Data Loss Prevention API, etc., if sensitive information is found, that portion is masked or replaced. For example, if "sales target" is determined to be sensitive information, it will be masked to something like "We achieved our target."

[1554] Furthermore, the server uses emotion recognition to identify the user's emotions from the audio data and adds that information to the text data. Google Cloud's Natural Language API, among others, is used for this emotion recognition. For example, if the user's emotions (such as joy or accomplishment) are detected from the tone and pitch of their voice, that emotion information is added to the text data as a label.

[1555] Once the final text data is generated, the server sends it back to the terminal. The terminal displays the returned text data on the user's screen in real time. This allows the user to see subtitles of what was said during the meeting, including the emotions expressed. For example, it might display something like, "The next project starts next month [excited]."

[1556] Specific example

[1557] If user A says "The next project starts next month" during a Zoom meeting, this audio is captured on the device, split, and sent to the server. The server feeds this audio into the Google Speech-to-Text API to generate text data that reads "The next project starts next month." This data is then checked by the Data Loss Prevention API and determined to be free of confidential information. Furthermore, an emotion recognition engine adds emotion information, such as "joy." This final text data is then sent to the device and displayed in real time.

[1558] In another scenario, User B says in Japanese, "We have achieved our sales targets." This audio is captured on the device, split, and sent to the server. The server uses the Google Speech-to-Text API and the DeepL API to translate this audio into the English text, "We have achieved our sales targets." The Data Loss Prevention API then determines that no sensitive information is included. An emotion recognition engine adds the emotion "sense of accomplishment." The resulting text data is sent to the device and displayed in real time.

[1559] Example of a prompt

[1560] Speech recognition prompt:

[1561] Please convert "The next project will start next month" into text.

[1562] Translation prompt:

[1563] Please translate "We achieved our sales target" into English.

[1564] Emotion recognition prompt:

[1565] Identify the user's emotions from the audio data and add annotations to the results.

[1566] Thus, the system of the present invention improves user communication by recognizing, translating, filtering, and recognizing the emotions of speech during meetings in real time, and displaying it as subtitles.

[1567] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1568] Specific flow of the system program's processing

[1569] Step 1:

[1570] Collection of audio data

[1571] The device uses an audio input device (microphone) to capture the user's spoken audio in real time. For example, if a user says "The next project starts next month" during a Zoom meeting, this audio will be captured.

[1572] Input: User's spoken audio.

[1573] Specific operation: The captured audio data is divided into chunks of one second each. The terminal then sends this chunked audio data to the next processing step.

[1574] Step 2:

[1575] Sending audio data

[1576] The terminal sends chunked audio data to the server in real time.

[1577] Input: Audio data chunked every second.

[1578] Specific operation: The device uploads this data to the server via the internet. For example, real-time transfer using WebSocket is a possible approach.

[1579] Step 3:

[1580] Speech recognition

[1581] The server inputs the received audio data into an AI speech recognition model and converts it into text data. APIs such as Google Speech-to-Text are used for this purpose.

[1582] Input: Audio data sent from the device.

[1583] Output: Converted text data. (Example: "The next project starts next month.")

[1584] Specific operation: The server processes the audio chunks sequentially to generate high-precision text data. This data is then sent to subsequent processing steps.

[1585] Step 4:

[1586] translation

[1587] The server feeds the generated text data into the translation system as needed, and translates it into the desired language. The DeepL API is used for translation.

[1588] Input: Converted text data. (Example: "The next project starts next month.")

[1589] Output: Translated text data. (Example: "The next project will start from next month")

[1590] Specific operation: The server sends text data based on the configured target language and receives the translation result. This data is then sent to the next filtering step.

[1591] Step 5:

[1592] filtering

[1593] The server filters the translated text data and takes measures to detect sensitive information. Data Loss Prevention APIs, among others, are used.

[1594] Input: Translated text data. (Example: "The next project will start from next month")

[1595] Output: Filtered text data. (If there is confidential information, e.g., "The next project will start from next month")

[1596] Specific operation: The server scans the text data and detects keywords related to sensitive information. Those parts are then replaced or masked.

[1597] Step 6:

[1598] emotion recognition

[1599] The server uses emotion recognition to identify the user's emotions from the audio data and adds that information to the text data. Google Cloud's Natural Language API is used.

[1600] Input: User's voice data and filtered text data. (Example: "The next project will from next month")

[1601] Output: Text data with emotion labels. (Example: "The next project will from next month [joy]")

[1602] Specific operation: Emotional features such as tone and pitch are extracted from audio data, and the detected emotions (joy, anxiety, etc.) are added as labels to the text data.

[1603] Step 7:

[1604] Display subtitles

[1605] The server sends the final text data to the terminal and displays it on the user's screen in real time.

[1606] Input: Text data with emotion labels attached. (Example: "The next project will from next month [joy]")

[1607] Output: Subtitles displayed on the user's device. (Example: "The next project will be from next month [joy]")

[1608] Specific operation: The device displays this data in the caption box of the conferencing software. This allows users to see the content and sentiment information of what is being said in real time.

[1609] This series of processing steps enables a system that recognizes, translates, filters, and recognizes the sentiment of audio data during a meeting in real time, and displays it to the user immediately.

[1610] (Application Example 2)

[1611] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1612] Traditional meeting and customer service systems had limited voice recognition and translation capabilities, making them inadequate for real-time communication, multilingual support, and the protection of confidential information. Furthermore, the lack of means to recognize and provide real-time information about the emotions of customers and meeting participants resulted in a decline in communication quality. Additionally, the reliability of translation results and the protection of confidential information were insufficient, necessitating appropriate solutions.

[1613] In Application Example 2, the specific processing performed by the specific processing unit 290 of the data processing device 12 is realized by the following means. In this invention, the server includes means for collecting voice data, means for transmitting the collected voice data to the server in real time, voice recognition means for converting the transmitted voice data into text data, translation means for translating the converted text data into different languages, filtering means for detecting, masking, or replacing confidential information from the translated text data, emotion recognition means for identifying emotional states from the voice data and adding that information to the text data, and means for transmitting the final text data to a receiving terminal and displaying it in real time. This enables voice recognition, translation, protection of confidential information, and real-time display including emotion recognition.

[1614] "Audio data" refers to a data format in which sound is recorded electronically and can be analyzed and converted through computer processing.

[1615] "Means of collection" refers to devices or systems that capture audio data in real time and store it in digital format.

[1616] "Means of sending to the server in real time" refers to communication methods for transferring collected audio data to the server without delay.

[1617] "Speech recognition means" refers to a software or hardware configuration for analyzing speech data and converting it into text data.

[1618] "Translation means" refers to software or services for converting text data into different languages.

[1619] A "filtering method" is an algorithm that automatically detects confidential information from text data and masks or replaces that portion.

[1620] "Emotion recognition means" refers to software or models that detect emotions from audio or text data and add that information.

[1621] "Final text data" refers to the character data after processing is complete, including all processing such as speech recognition, translation, filtering, and sentiment recognition.

[1622] A "receiving terminal" is a device used to display the final text data, and includes smartphones, smart glasses, and other similar devices.

[1623] The embodiments for carrying out the present invention will be described in detail below.

[1624] This system processes voice data collection, speech recognition, translation, filtering, sentiment recognition, and real-time display as a series of processes. The voice spoken by the user is first captured in real time by a dedicated device (such as smart glasses or a smartphone). Suitable hardware includes a microphone and a high-speed central processing unit (CPU).

[1625] After the collection of audio data is complete, the collected data is transmitted to the server in real time. High-speed communication lines such as Wi-Fi or 4G / 5G are used for this transmission. The server first appropriately divides (chunks) the received audio data into digital format, and then divides it into appropriate sizes according to the amount of data.

[1626] The segmented audio data is then converted into text data using speech recognition software such as the Google Cloud Speech-to-Text API or IBM Watson Speech to Text. The converted text data is then translated into the specified language using the Google Cloud Translation API, if necessary.

[1627] The translated text data is further filtered to automatically detect and mask or replace personal or confidential information. This minimizes the risk of information leakage.

[1628] Subsequently, IBM Watson Tone Analyzer or a similar emotion recognition engine is used to identify the user's emotional state from the text data. Emotional information is added to the text data, and the final text data including this information is generated.

[1629] This final text data is sent back to the device and displayed to the user in real time. By displaying it on a screen such as smart glasses or a smartphone, the user can see not only what they have said, but also the emotions and important information contained within that statement in real time.

[1630] Specific example:

[1631] The following prompt messages are a concrete example of how this system can be applied to a customer service system in a physical store.

[1632] "Development of a system that recognizes emotions from customer statements, performs necessary translations, and displays them in real time on smart glasses."

[1633] Follow these steps to develop a system that recognizes emotions from customer speech, performs necessary translations, and displays them in real time on smart glasses.

[1634] 1. Capture customer voices in real time and convert them to text.

[1635] 2. If necessary, translate the text into the specified language.

[1636] 3. Detect and mask confidential information from the text.

[1637] 4. Recognize customer emotions from voice and add that information to text.

[1638] 5. Display the final text data on the smart glasses' display in real time.

[1639] This system facilitates multilingual support and enables more effective customer service by recognizing customer emotions. Furthermore, it provides a user-friendly environment while appropriately protecting confidential information.

[1640] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1641] Step 1:

[1642] The device captures the user's spoken voice in real time and converts it into a digital format. Specifically, the device's microphone collects the voice and saves it as an audio file. The input to this process is the user's voice data, and the output is a digital audio file.

[1643] Step 2:

[1644] The terminal transmits captured audio data to the server in real time. Specifically, the terminal uses a high-speed communication line such as Wi-Fi or 4G / 5G to divide the audio data into data packets and send them to the server. The input to this process is an audio file, and the output is the audio data sent to the server.

[1645] Step 3:

[1646] The server divides the received audio data into appropriate-sized chunks. Specifically, the server divides the audio file at regular time intervals (e.g., 1 second) and saves each chunk separately. The input to this process is the transmitted audio data, and the output is the divided audio chunks.

[1647] Step 4:

[1648] The server inputs the divided audio chunks into a speech recognition system and converts them into text data. Specifically, the server calls the Google Cloud Speech-to-Text API to convert the audio data into text data. The input to this process is audio chunks, and the output is text data.

[1649] Step 5:

[1650] The server translates the converted text data into a different language. Specifically, the server uses the Google Cloud Translation API to translate the text data into the specified target language. The input to this process is the text data, and the output is the translated text data.

[1651] Step 6:

[1652] The server detects sensitive information from translated text data and masks or replaces it. Specifically, the server uses regular expressions and machine learning models to identify personal and sensitive information and replaces the detected information with masking characters such as "". The input to this process is translated text data, and the output is filtered text data.

[1653] Step 7:

[1654] The server inputs filtered text data into an emotion recognition system, identifies the user's emotional state, and adds that information to the text data. Specifically, the server uses IBM Watson Tone Analyzer to perform emotion analysis and adds the results to the original text data. The input to this process is filtered text data, and the output is text data with added emotion information.

[1655] Step 8:

[1656] The server sends the final text data to the receiving terminal and displays it in real time. Specifically, the server sends the generated text data to the terminal as real-time subtitles, which the terminal displays on its screen. The input to this process is text data with added emotional information, and the output is real-time subtitles displayed on the user's screen.

[1657] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1658] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1659] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[1660] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1661] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[1662] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[1663] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[1664] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[1665] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[1666] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[1667] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[1668] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[1669] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[1670] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1671] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[1672] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[1673] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[1674] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[1675] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[1676] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[1677] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[1678] The following is further disclosed regarding the embodiments described above.

[1679] (Claim 1)

[1680] In a conference system,

[1681] Means of collecting audio data,

[1682] A means of transmitting the collected audio data to the server in real time,

[1683] A speech recognition means for converting transmitted audio data into text data,

[1684] A translation method for translating converted text data into a different language,

[1685] A filtering means for detecting and masking or replacing confidential information from translated text data,

[1686] A means of sending the final text data to the receiving terminal and displaying it in real time,

[1687] A system that includes this.

[1688] (Claim 2)

[1689] The system according to claim 1, further comprising means for dividing audio data into appropriate chunks.

[1690] (Claim 3)

[1691] The system according to claim 1, wherein the above-mentioned speech recognition means uses an artificial intelligence model.

[1692] "Example 1"

[1693] (Claim 1)

[1694] Means of collecting audio data,

[1695] A means of transmitting the collected audio data to the server in real time,

[1696] A speech recognition means for converting transmitted audio data into text data,

[1697] A translation method for translating converted text data into a different language,

[1698] A filtering means for detecting and masking or replacing confidential information from translated text data,

[1699] A means of sending text data to a server and displaying it on a receiving terminal,

[1700] A means of displaying text data in real time,

[1701] A means of analyzing audio data and dividing it into appropriate sizes,

[1702] A system that includes this.

[1703] (Claim 2)

[1704] The system according to claim 1, wherein the speech recognition means uses an artificial intelligence model.

[1705] (Claim 3)

[1706] The system according to claim 1, further comprising means for compressing and transmitting audio data divided into appropriate chunks.

[1707] "Application Example 1"

[1708] (Claim 1)

[1709] In conference systems and in-vehicle communication systems,

[1710] Means of collecting audio data,

[1711] A means of transmitting the collected audio data to the server in real time,

[1712] A speech recognition means for converting transmitted audio data into text data,

[1713] A translation method for translating converted text data into a different language,

[1714] A filtering means for detecting and masking or replacing confidential information from translated text data,

[1715] A means of sending the final text data to the receiving terminal and displaying it in real time,

[1716] A means of displaying text data in real time on an in-vehicle display,

[1717] A system that includes this.

[1718] (Claim 2)

[1719] The system according to claim 1, further comprising means for dividing audio data into appropriate chunks.

[1720] (Claim 3)

[1721] The system according to claim 1, wherein the above-mentioned speech recognition means uses a generating AI model.

[1722] "Example 2 of combining an emotion engine"

[1723] (Claim 1)

[1724] Means of collecting audio data,

[1725] A means of transmitting the collected audio data to the server in real time,

[1726] A speech recognition means for converting transmitted audio data into text data,

[1727] A translation method for translating converted text data into a different language,

[1728] A filtering means for detecting and masking or replacing confidential information from translated text data,

[1729] An emotion recognition means that recognizes emotions and adds that information to text data,

[1730] A means of sending the final text data to the receiving terminal and displaying it in real time,

[1731] A system that includes this.

[1732] (Claim 2)

[1733] The system according to claim 1, further comprising means for dividing audio data into appropriate chunks.

[1734] (Claim 3)

[1735] The system according to claim 1, wherein the above-mentioned speech recognition means uses a generated artificial intelligence model.

[1736] "Application example 2 when combining with an emotional engine"

[1737] (Claim 1)

[1738] Means of collecting audio data,

[1739] A means of transmitting the collected audio data to the server in real time,

[1740] A speech recognition means for converting transmitted audio data into text data,

[1741] A translation method for translating converted text data into a different language,

[1742] A filtering means for detecting and masking or replacing confidential information from translated text data,

[1743] An emotion recognition means that identifies emotional states from audio data and adds that information to text data,

[1744] A means of sending the final text data to the receiving terminal and displaying it in real time,

[1745] A system that includes this.

[1746] (Claim 2)

[1747] The system according to claim 1, further comprising means for dividing audio data into appropriate chunks.

[1748] (Claim 3)

[1749] The system according to claim 1, wherein the above-mentioned speech recognition means uses an artificial intelligence model. [Explanation of symbols]

[1750] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. In a conference system, Means of collecting audio data, A means of transmitting the collected audio data to the server in real time, A speech recognition means for converting transmitted audio data into text data, A translation method for translating converted text data into a different language, A filtering means for detecting and masking or replacing confidential information from translated text data, A means of sending the final text data to the receiving terminal and displaying it in real time, A system that includes this.

2. The system according to claim 1, further comprising means for dividing audio data into appropriate chunks.

3. The system according to claim 1, wherein the above-mentioned speech recognition means uses an artificial intelligence model.

Images