System

Abstract

A system is provided.SOLUTION: A system comprising: means for converting input speech into text; means for regenerating the generated text into a meaningful sentence; and means for converting the regenerated text into speech and outputting the speech.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Currently, television broadcasts do not provide sufficient information to people who have difficulty accessing text or audio information, such as the hearing impaired, foreign language learners, visually impaired people, and busy people. This poses a challenge, making it difficult for these people to fully understand the content of television broadcasts. [Means for solving the problem]

[0005] To solve these problems, the present invention provides a system including a means for converting input speech into text, a means for regenerating the generated text into meaningful sentences, and a means for converting the regenerated text into speech and outputting it. Specifically, the system converts input speech into text using speech recognition technology, regenerates the generated text into meaningful sentences using generative artificial intelligence, and converts the regenerated text into speech using text-to-speech technology. Furthermore, the system includes a means for capturing input speech in real time and converting it into text, a means for displaying the generated text in a scrolling format on a display device, and a means for providing the output speech to a user using an audio output device, thereby providing a comprehensive and inclusive listening experience.

[0006] The "means for converting input voice into text" is a function that uses voice recognition technology to convert voice data provided by the user into text information.

[0007] The "means for regenerating generated text into meaningful sentences" is a function that uses generative artificial intelligence to reorganize initial text data into meaningful sentences in accordance with the context.

[0008] The "means for converting the reproduced text into speech and outputting it" is a function that uses text-to-speech conversion technology to convert the generated text into speech data and provide it to the user.

[0009] "Speech recognition technology" is a technology for analyzing input voice data and converting it into corresponding text data.

[0010] "Generative AI" is an AI technique for regenerating initial text data into meaningful sentences based on context and grammar.

[0011] "Text-to-speech technology" is a technology for converting text data into natural-sounding speech.

[0012] The "means for displaying in a scrolling format on a display device" is a function for continuously displaying the generated text on the screen so that the user can visually confirm it.

[0013] The "means for providing the user with audio output using an audio output device" is a function for playing back the generated audio data to the user via an audio output device such as a speaker or headphones.

[0014] "Means for capturing and converting to text in real time" refers to a function that captures audio data on the spot and instantly converts it into text data. [Brief explanation of the drawings]

[0015] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11]FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0016] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0017] First, the terms used in the following description will be explained.

[0018] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0019] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0020] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0021] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0022] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0023] [First embodiment]

[0024] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0025] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0026] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0027] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0028] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0029] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0030] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0031] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0032] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0033] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0034] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0035] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0036] A specific example of the configuration of a system for implementing the present invention is shown below.

[0037] System Overview

[0038] This system converts speech to text, regenerates the text, and converts the regenerated text back into speech for output, providing a comprehensive viewing experience. The specific processing flow and functions of this system are described below.

[0039] System configuration

[0040] This system consists of three elements: a server, a terminal, and a user. The system operates according to the following procedure.

[0041] Audio collection and text conversion

[0042] 1. Collecting voice input

[0043] The user inputs the audio of the television broadcast through the microphone of the device, for example, by issuing a voice command such as "display subtitles for the news" or by speaking the actual content of the television news broadcast.

[0044] The terminal captures this audio in real time and transmits the audio data as is to the server.

[0045] 2. Speech-to-text

[0046] The server converts the received voice data into text data using speech recognition technology. For example, a voice saying "Show me the subtitles for the news" is converted into text "Show me the subtitles for the news."

[0047] Text semantic generation and refinement

[0048] 3. Text Analysis and Generation

[0049] The server uses generative artificial intelligence to regenerate meaningful sentences based on the initial text generated by speech recognition technology, taking into account context and grammar to generate natural-sounding sentences. For example, the generated sentence might read, "In the news: The stock market rose significantly today."

[0050] 4. Text Tweaks

[0051] The server checks the generated text for grammar and punctuation and makes corrections if necessary. The final text is "In the news: The stock market rose significantly today."

[0052] Text-to-speech conversion and output

[0053] 5. Text to speech conversion

[0054] The server converts the final text into audio data using text-to-speech technology, and the resulting audio is played back as a natural voice.

[0055] 6. Text and audio transmission and output

[0056] The server transmits the generated voice data and the final text to the terminal.

[0057] The device displays subtitles on the screen, such as "In the news: The stock market saw a big jump today," while simultaneously playing audio data, allowing the user to consume both audio and text information.

[0058] Specific examples

[0059] For example, if a user says "Display subtitles for the news" in front of a TV, the system operates as follows.

[0060] 1. The user says, "Show me closed captions for the news."

[0061] 2. The device captures the audio and sends it to the server.

[0062] 3. The server uses speech recognition technology to convert the text into "Show me the news subtitles."

[0063] 4. The server uses generative artificial intelligence to recreate the broadcast: "In the news: The stock market rose significantly today."

[0064] 5. The server then converts this text into audio data using text-to-speech technology.

[0065] 6. The server sends the generated voice data and text to the terminal.

[0066] 7. The device will display the caption "In the news: The stock market saw a big jump today" on the screen and simultaneously play the audio.

[0067] In this way, the system converts speech to text, regenerates the text into meaningful sentences, and converts the regenerated text into speech for output, providing a comprehensive listening experience.

[0068] The processing flow will be explained below.

[0069] Step 1:

[0070] The user speaks the audio of the television broadcast into the device's microphone, for example, saying "display the news subtitles" or the actual content of the television news broadcast.

[0071] Step 2:

[0072] The terminal captures the user's voice in real time through a microphone and acquires it as voice data.

[0073] Step 3:

[0074] The device transmits the acquired voice data to a server via the Internet.

[0075] Step 4:

[0076] The server converts the received voice data into text data using voice recognition technology. For example, it converts the voice "Show me the subtitles for the news" into the text "Show me the subtitles for the news."

[0077] Step 5:

[0078] The server uses generative artificial intelligence to regenerate the initial text obtained from the speech recognition technology into a meaningful sentence based on the context, for example, "Today's news is that the stock market has risen significantly."

[0079] Step 6:

[0080] The server checks the generated text for grammar, punctuation, etc. and makes any necessary adjustments. The final text is "Today's news is that the stock market has risen significantly."

[0081] Step 7:

[0082] The server converts the final text data into audio data using text-to-speech technology, and the generated audio can be played back as a natural voice.

[0083] Step 8:

[0084] The server transmits the generated voice data and the final text to the terminal.

[0085] Step 9:

[0086] The terminal displays the final text "Today's news is that the stock market has risen significantly" in scrolling format on the screen.

[0087] Step 10:

[0088] The terminal reproduces the transmitted audio data to the user through an audio output device (speaker or headphones), allowing the user to use both audio information and text information.

[0089] Example 1

[0090] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0091] Conventional speech recognition systems simply convert speech to text without correcting the meaning or grammar of the text. This can result in unnatural and difficult-to-understand text for users. Furthermore, systems that convert the converted text back into speech and output it are limited. The present invention aims to solve these problems by providing a comprehensive system that converts speech to text, regenerates the text into natural-sounding sentences, and then re-outputs it as speech.

[0092] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0093] In this invention, the server includes means for converting input speech into text, means for regenerating the generated text into meaningful sentences, means for converting the regenerated text into speech and outputting it, means for capturing input speech in real time, means for checking the generated text with a grammar checker and correcting it if necessary, and means for providing the generated speech data and final text by screen display and audio playback, thereby enabling users to use natural and easy-to-understand text and speech information in real time.

[0094] "Speech recognition technology" is a technology that analyzes input voice data and converts it into text data.

[0095] "Generative AI" is an AI technology that generates natural-looking sentences based on input initial text, taking into account context and grammar.

[0096] "Text-to-speech technology" is a technology that converts text data into voice data and outputs it as natural voice.

[0097] "Capturing in real time" means that input audio data can be processed immediately as it is.

[0098] A "grammar checker" is a tool or technique for checking the grammar and punctuation of generated text and correcting errors.

[0099] "Displaying on screen" means displaying the generated text on a display device such as a monitor or screen.

[0100] "Audio playback" refers to playing back the generated audio data through an audio output device such as a speaker.

[0101] "Means" refers to a method, technique, or device used to achieve a particular purpose.

[0102] A "server" is a computer system connected to a network that processes requests from multiple terminals and provides data.

[0103] "Terminal" means a device that allows a user to input voice and display / play back the generated text and voice.

[0104] "User" refers to a person using the system to provide voice input.

[0105] A "network" is a communications infrastructure that allows multiple computer systems to communicate with each other.

[0106] An "input prompt" refers to the text of instructions or questions given to a generative artificial intelligence.

[0107] System Overview

[0108] The system of the present invention converts speech into text, regenerates the text into meaningful sentences, and finally converts the regenerated text into speech and outputs it. The system consists of three elements: a server, a terminal, and a user, and is realized using the following hardware and software.

[0109] Hardware and software used

[0110] 1. Hardware

[0111] Terminal: A device equipped with a microphone, display, and speaker that allows a user to input voice and output generated text and voice.

[0112] Server: Equipped with the computing resources to perform speech recognition, generative artificial intelligence, and text-to-speech conversion.

[0113] 2. Software

[0114] Speech recognition technology: Using Google Cloud Speech-to-Text, etc., voice data is converted into text data.

[0115] Generative artificial intelligence: Using models such as OpenAI GPT-4, initial text is generated into meaningful sentences.

[0116] Grammar checker: Checks and corrects the grammar of the generated text using APIs such as Grammarly.

[0117] Text-to-speech technology: Using technologies such as Amazon Polly, text data is converted into natural-sounding speech data.

[0118] System Operation

[0119] 1. The user stands in front of the device and gives a voice command, such as "Show subtitles for the news." The device's microphone captures this voice in real time and sends the voice data to the server.

[0120] 2. The server converts the received voice data into text using Google Cloud Speech-to-Text. At this time, the voice command "Show news subtitles" is converted into text data "Show news subtitles."

[0121] 3. The server uses generative artificial intelligence (OpenAI GPT-4) to regenerate the text data into natural-sounding sentences. For example, if a user is watching the news on TV, the initial text "Show me the news subtitles" is generated into a detailed sentence: "News: The stock market rose significantly today." The input prompt for the generative AI model will be in the following text format:

[0122] Example prompt: "Regenerate this text into a meaningful sentence. Original text: 'In the news: The stock market rose significantly today.'"

[0123] 4. The server checks the generated text with a grammar checker (Grammarly API) and corrects any errors or unnatural points to generate the final text.

[0124] 5. The server converts the final text into speech using Amazon Polly, generating a natural-sounding speech, for example, "In the news: The stock market rose significantly today."

[0125] 6. The server sends the generated voice data and the final text to the terminal, which displays "In the news: The stock market rose significantly today" on the screen and plays the voice at the same time.

[0126] Specific examples

[0127] As a specific example, the system operation when the user utters "Display subtitles for the news" will be described.

[0128] 1. The user says, "Show me closed captions for the news."

[0129] 2. The device captures the audio and sends it to the server.

[0130] 3. The server converts the audio data into text.

[0131] 4. The server uses the generative AI model to regenerate the initial text into detailed sentences.

[0132] 5. The server checks the generated text with a grammar checker and corrects it.

[0133] 6. The server converts the final text into audio data.

[0134] 7. The server sends the voice data and text to the device.

[0135] 8. The device displays the final text and plays the audio.

[0136] In this way, the system of the present invention can convert speech into text and speech in real time, providing information to the user in a natural and easy-to-understand manner.

[0137] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0138] Step 1:

[0139] The process begins when a user speaks a command in front of the device. For example, they might say, "Display subtitles for the news." This voice is captured by the device's microphone and saved as audio data in real time. The input is audio data, and the output is the audio data saved on the device. The device then prepares to send this audio data to the server via the network.

[0140] Step 2:

[0141] The device transmits the captured audio data to the server via the network. The input here is the audio data stored on the device, and the output is the audio data to be transmitted to the server. The device establishes a network connection and executes communication processing to transfer the audio data to the server.

[0142] Step 3:

[0143] The server converts the voice data received from the device into text data using voice recognition technology such as Google Cloud Speech-to-Text. The input is the voice data received by the server, and the output is text data. The server analyzes the voice data and performs the specific operation of calling a voice recognition API to convert it into text.

[0144] Step 4:

[0145] The server uses generative artificial intelligence (e.g., OpenAI GPT-4) to regenerate the initial text into a meaningful sentence. The input is text data obtained by speech recognition technology, and the output is the regenerated sentence. The specific format of the prompt sentence is, "Please regenerate this text into a meaningful sentence. Original text: 'News: The stock market rose significantly today.'" The server sends the prompt sentence to the generative AI model and obtains the generated sentence.

[0146] Step 5:

[0147] The server checks the generated text with a grammar checker (e.g., Grammarly API) and corrects grammar and punctuation. The input is the text generated by the generative AI model, and the output is the final text after corrections. The server sends the text to the grammar checker, receives correction feedback, and incorporates it.

[0148] Step 6:

[0149] The server converts the final corrected text into audio data using text-to-speech technology such as Amazon Polly. The input is the final text and the output is audio data. The server calls the text-to-speech API to convert the text into audio and obtains the generated audio data.

[0150] Step 7:

[0151] The server sends the generated voice data and final text to the terminal. The input is the generated voice data and final text, and the output is the data to be sent to the terminal. The server performs communication processing to transfer the voice data and text to the terminal via the network.

[0152] Step 8:

[0153] The terminal displays the final text received from the server on the screen and simultaneously plays the audio data from the speaker. The input is the audio data and text data received from the server, and the output is the text display on the screen and audio playback from the speaker. Specifically, the terminal outputs the final text to the display and plays the audio data on the audio device.

[0154] By performing the above steps, the system of the present invention can convert speech into text and speech in real time, and provide information to the user in a natural and easy-to-understand format.

[0155] (Application example 1)

[0156] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0157] Conventional content viewing systems lack a mechanism for providing real-time subtitles or summaries of the content users are watching, which results in a lack of an improved viewing experience for users, making it difficult to understand specific scenes or easily check the introduction of the next episode through audio and text.

[0158] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0159] In this invention, the server includes means for converting input speech into text, means for regenerating the generated text into meaningful sentences, means for converting the regenerated text into speech and outputting it, means for displaying the generated text on a display device, means for providing the output speech using a speech output device, means for summarizing the next content based on the generated text, and communication means for transmitting the generated text and speech data to a user terminal, thereby enabling the user to check subtitles and content summaries in both speech and text in real time.

[0160] The "means for converting input voice into text" refers to a means for analyzing voice data input by a user and converting the content of the voice data into text data of the corresponding sentence.

[0161] The "means for regenerating generated text into meaningful sentences" refers to a means for processing the text generated by the initial speech recognition to reconstruct it into natural sentences based on context and grammar.

[0162] The "means for converting the regenerated text into speech and outputting the speech" refers to a means for converting the regenerated text data into speech data and performing processing to output the speech data to the user.

[0163] "Means for displaying the generated text on a display device" refers to means for transmitting the generated text data to a display device such as a display or screen in order to visually display it, and displaying it in a form that can be confirmed by the user.

[0164] The "means for providing audio output using an audio output device" refers to a means for allowing a user to listen to the generated audio data via an audio output device such as a speaker or headphones.

[0165] The "means for summarizing the next content based on the generated text" is a processing means for briefly summarizing the next episode or important content based on the initial input speech and the regenerated text.

[0166] "Communication means for transmitting the generated text and voice data to the user's terminal" refers to a means including communication technology for transmitting the generated text and voice data from the server to a terminal such as the user's smartphone or head-mounted display.

[0167] System Overview

[0168] This invention is a system that provides subtitles and summaries of content in real time in response to user voice commands while viewing content. This system consists of three main elements: a server, a terminal, and a user, and these elements work in conjunction with each other.

[0169] Hardware and Software Configuration

[0170] server:

[0171] Speech recognition engine used: Google Speech-to-Text API

[0172] Generative artificial intelligence model used: OpenAI GPT model

[0173] Text-to-speech engine used: Amazon Polly

[0174] Communication method: Internet communication

[0175] Device:

[0176] Audio input device: Microphone (built into smartphone or head-mounted display)

[0177] Audio output device: Speakers or headphones

[0178] Display devices: smartphone and head-mounted display screens

[0179] Processing flow

[0180] When a user uses a device to give voice commands, the device's microphone captures the voice and sends it to a server over the Internet. The server converts the voice into text using a speech recognition engine (Google Speech-to-Text API). It then uses a generative artificial intelligence model (OpenAI GPT model) to regenerate the text into natural-sounding sentences appropriate to the context. The generated text is then converted into audio data by a text-to-speech engine (Amazon Polly) and sent back to the device over the Internet. Finally, the device displays the text on the screen and plays the audio through speakers or headphones.

[0181] Specific example explanation

[0182] Example 1: Displaying subtitles

[0183] While watching a movie, a user says, "Show subtitles." The device's microphone captures the audio and sends it to the server. The server converts the audio to text, regenerates it, converts it back to audio, and sends it to the device. The device displays on the screen, "Movie content: The protagonist sets off on a new adventure." and simultaneously plays the audio.

[0184] Example 2: Next episode introduction

[0185] While watching a drama, a user says, "Please tell me about the next episode." The device's microphone captures the audio and sends it to the server. The server converts the audio into text and regenerates it based on the context. The generated text reads, "In the next episode, the protagonist sets out on a new adventure." This is converted into audio and sent to the device. The device displays "In the next episode, the protagonist sets out on a new adventure" on the screen and simultaneously plays it aloud.

[0186] Prompt Sentence Examples

[0187] User voice: "Please introduce the next episode."

[0188] Translated text: "Please introduce the next episode"

[0189] Example of a generative AI model input prompt: "Please explain the content of the next episode."

[0190] Generated output: "In the next episode, our protagonist sets off on a new adventure, facing various obstacles and meeting new allies."

[0191] In this way, the system converts speech to text and displays and speaks the reproduced text, providing a user with an enhanced viewing experience.

[0192] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0193] Step 1:

[0194] The user gives a voice command. For example, the user says, "Please introduce the next episode." This becomes the voice input data.

[0195] Step 2:

[0196] The device's microphone captures the voice input data, which is then sent by the device to the server.

[0197] Step 3:

[0198] The server receives the voice data and converts it into text data using a speech recognition engine (Google Speech-to-Text API). The input in this step is voice data, and the output is text data.

[0199] Step 4:

[0200] The server uses a generative artificial intelligence model (OpenAI GPT model) to regenerate the converted text data into meaningful sentences. In this step, it takes the text data as input and generates natural-looking sentences that are appropriate for the context. The output is the regenerated text.

[0201] Step 5:

[0202] The server converts the regenerated text data into audio data using a text-to-speech engine (Amazon Polly). The input is the regenerated text, and the output is audio data.

[0203] Step 6:

[0204] The server sends the generated voice data and text data to the terminal, which outputs the voice data and text data.

[0205] Step 7:

[0206] The text data received by the terminal is displayed on the screen. At the same time, the audio data is played back using the terminal's audio output device (speaker or headphones). This allows the user to see the text on the screen and hear the content aloud. The input is text data and audio data, and the output is screen display and audio playback.

[0207] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0208] A specific example of the configuration of a system for implementing the present invention is shown below.

[0209] System Overview

[0210] This system not only converts speech to text, regenerates the text, and converts the regenerated text back into speech for output, but also recognizes the user's emotions and reflects them in the output content, providing a more comprehensive and personalized viewing experience. The specific processing flow and functions of this system are described below.

[0211] System configuration

[0212] This system consists of three elements: a server, a terminal, and a user. The system operates according to the following procedure.

[0213] Audio collection and text conversion

[0214] 1. Collecting voice input

[0215] The user speaks the audio of the television broadcast into the microphone of the terminal, for example, "display the subtitles of the news" or the actual content of the television news broadcast.

[0216] The terminal captures this audio in real time and transmits the audio data as is to the server.

[0217] 2. Speech-to-text

[0218] The server converts the received voice data into text data using voice recognition technology. For example, it converts the voice "Show me the subtitles for the news" into the text "Show me the subtitles for the news."

[0219] Text semantic generation and refinement

[0220] 3. Text Analysis and Generation

[0221] The server uses generative artificial intelligence to regenerate meaningful sentences based on the initial text generated by speech recognition technology. At this time, the server takes into account context and grammar to generate natural-sounding sentences. For example, it generates a sentence like, "Today's news is that the stock market has risen significantly."

[0222] 4. Text Tweaks

[0223] The server checks the generated text for grammar, punctuation, etc. and makes any necessary adjustments. The final text is "Today's news is that the stock market has risen significantly."

[0224] User Emotion Recognition

[0225] 5. Emotion collection and analysis

[0226] The device uses an emotion recognition engine to analyze the user's facial expressions and tone of voice during voice input. This analysis is performed in real time and emotion data is generated.

[0227] The server receives the emotion data and reflects it in the generated text and voice.

[0228] Text-to-speech conversion and output

[0229] 6. Text to speech conversion

[0230] The server then converts the final text data into audio data using text-to-speech technology, with the tone and speed of the generated audio adjusted to match the user's emotions.

[0231] 7. Text and audio transmission and output

[0232] The server transmits the generated voice data and the final text to the terminal.

[0233] The terminal displays the final text "Today's news is that the stock market has risen significantly" in scrolling format on the screen.

[0234] The device then plays the audio data back to the user through an audio output device (speaker or headphones). The audio is tailored to the user's emotions, providing a more personalized listening experience.

[0235] Specific examples

[0236] For example, if a user stands in front of a TV and says, "Display subtitles for the news," with a surprised expression on their face, the system operates as follows.

[0237] 1. The user looks surprised while saying, "Show me the news subtitles."

[0238] 2. The device captures the voice and facial expressions and sends the voice and emotion data to the server.

[0239] 3. The server uses speech recognition technology to convert the text into "Show me the news subtitles."

[0240] 4. The server uses generative artificial intelligence to recreate the broadcast: "In the news: The stock market rose significantly today."

[0241] 5. The server adjusts the text and tone of the voice based on the emotional data.

[0242] 6. The server sends the final text and adjusted audio data to the device.

[0243] 7. The device displays the caption "In the news: The stock market rose significantly today" on the screen and plays a voiceover in a calibrated tone of surprise.

[0244] In this way, the system converts speech to text and then converts the regenerated text into speech that matches the user's emotions, providing a personalized listening experience.

[0245] The processing flow will be explained below.

[0246] Step 1:

[0247] The user speaks the audio of the television broadcast into the device's microphone, for example, saying "display the news subtitles" or the actual content of the television news broadcast.

[0248] Step 2:

[0249] The device captures the user's voice in real time through a microphone and acquires it as voice data, while also collecting emotional information such as the user's facial expressions and tone of voice.

[0250] Step 3:

[0251] The device transmits the acquired voice data and emotion data to a server via the Internet.

[0252] Step 4:

[0253] The server converts the received voice data into text data using voice recognition technology. For example, it converts the voice "Show me the subtitles for the news" into the text "Show me the subtitles for the news."

[0254] Step 5:

[0255] The server uses generative artificial intelligence to regenerate meaningful sentences based on the initial text generated by speech recognition technology. At this time, it takes into account context and grammar to generate natural sentences. For example, the generated sentence might be, "Today's news is that the stock market has risen significantly."

[0256] Step 6:

[0257] The server checks the generated text for grammar, punctuation, etc. and makes any necessary adjustments. The final text is "Today's news is that the stock market has risen significantly."

[0258] Step 7:

[0259] The server analyzes the received emotional data to recognize the user's emotional state, which includes facial expressions such as joy, surprise, and sadness, as well as tone of voice.

[0260] Step 8:

[0261] The server adjusts the text and audio output depending on the user's emotion. For example, if the emotion is "surprise," it will emphasize the text, raise the tone of the audio, and increase the playback speed.

[0262] Step 9:

[0263] The server converts the final text into audio data using text-to-speech technology, and the generated audio is adjusted in tone and speed to match the user's emotions.

[0264] Step 10:

[0265] The server sends the adjusted voice data and the final text to the terminal.

[0266] Step 11:

[0267] The terminal displays the final text "Today's news is that the stock market has risen significantly" in scrolling format on the screen.

[0268] Step 12:

[0269] The device then plays the audio data back to the user through an audio output device (speaker or headphones). The audio is tailored to the user's emotions, providing a more personalized listening experience.

[0270] Example 2

[0271] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0272] Conventional speech recognition systems often convert input speech into text and output the text as is. This makes it difficult to provide a personalized listening experience that reflects the user's emotions. Furthermore, the generated text often lacks proper grammar and tone adjustment, which can detract from a natural conversational experience. Furthermore, the visual display and audio output are not integrated, preventing comprehensive information provision.

[0273] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0274] In this invention, the server includes means for converting input speech into text, means for regenerating the generated text into meaningful sentences, means for recognizing a user's emotion and adjusting the tone and speed of the regenerated text and speech based on the emotion, means for converting the regenerated text into speech and outputting it, means for displaying the final text in a scrolling format on a display device, and means for providing the output speech to the user using an audio output device, thereby enabling a natural conversation experience and a personalized listening and listening experience that is tailored to the user's emotion.

[0275] The "means for converting input speech into text" refers to a system or technology that analyzes speech data input by a user and converts it into corresponding text data.

[0276] A "means for regenerating generated text into meaningful sentences" is a technology or system that uses initial text data to reconstruct sentences that have natural meaning, taking into account context and grammar.

[0277] "Means for recognizing user emotions" refers to technology or a system that analyzes the user's facial expressions and tone of voice to understand their emotional state at that time.

[0278] The "means for converting the reproduced text into speech and outputting it" refers to a technology or system that analyzes text data, converts it into corresponding speech data, and plays it back to the user.

[0279] The "means for displaying in a scrolling format on a display device" refers to a technology or system for visually displaying text data in a format that moves continuously on the display device.

[0280] "Speech recognition technology" is a technology that analyzes a voice signal and generates corresponding text data.

[0281] A "generative AI model" is a model that uses artificial intelligence to generate meaningful sentences from given text data.

[0282] An "emotion recognition engine" is a technology or system that analyzes a user's facial expressions and tone of voice to identify their emotional state.

[0283] "Text-to-speech technology" is a technology that analyzes text data and generates speech data based on its content.

[0284] An "audio output device" is a device or system that reproduces audio data and allows the user to listen to it.

[0285] The system of the present invention converts voice input into text, then regenerates the text into meaningful sentences, and further recognizes the user's emotions to adjust the tone and speed of the generated text and speech, providing a more comprehensive and personalized listening experience. This system consists of three elements: a server, a terminal, and a user, and operates according to the following process flow.

[0286] Audio collection and text conversion

[0287] The user speaks the audio of the television broadcast into the microphone of the terminal, for example, saying "Display subtitles for the news."

[0288] The device captures this voice in real time and generates digital audio data, which is then immediately sent to the server.

[0289] The server converts the received voice data into text data using speech recognition technology (commonly known as a speech recognition engine). A major speech recognition technology is Google Cloud Speech-to-Text. The converted text data is used in the next step.

[0290] Text semantic generation and refinement

[0291] The server uses a generative AI model (commonly known as a language generation model) to regenerate meaningful sentences based on the initial text generated by speech recognition. This series of processes uses GPT-3 and other technologies. Specifically, based on the text obtained through speech recognition, "Show me the subtitles for the news," it generates a natural-sounding sentence that is in line with the context, "Today's news is that the stock market has risen significantly."

[0292] After the text is generated, the server checks it for grammar and punctuation and makes any necessary adjustments, resulting in the final text sounding natural: "Today's news is that the stock market has risen significantly."

[0293] User Emotion Recognition

[0294] The device uses an emotion recognition engine (commonly known as an emotion analysis system) to analyze the user's facial expressions and tone of voice when they input voice. This analysis is performed in real time, and user emotion data is generated. A major emotion recognition engine is the Microsoft Azure Emotion API.

[0295] The server receives the emotion data and reflects it in the generated text and voice. Specifically, it adjusts the tone of the text or speeds up the voice depending on the emotion of surprise.

[0296] Text-to-speech conversion and output

[0297] The server then converts the final text data into audio data using text-to-speech technology (commonly known as a text-to-speech engine). Amazon Polly is used for this conversion. The generated audio is adjusted in tone and speed to match the user's emotion. For example, if surprise is recognized, a voice will be generated saying, "Today's news is that the stock market has risen significantly," in a surprised tone.

[0298] The server sends the generated voice data and the final text to the terminal. The terminal displays the final text, "Today's news is that the stock market has risen significantly," in a scrolling format on the screen. The terminal also plays the sent voice data through an audio output device (e.g., speaker or headphones). This allows the news to be conveyed in a tone that corresponds to the emotion.

[0299] Specific example explanation

[0300] The operation when a user is in front of a television and says "Display subtitles for the news" while making a surprised expression is as follows.

[0301] 1. The user looks surprised while saying, "Show me the news subtitles."

[0302] 2. The device captures the voice and facial expressions and sends the voice and emotion data to the server.

[0303] 3. The server uses speech recognition technology to convert the audio "Show me the news subtitles" into the text "Show me the news subtitles."

[0304] 4. The server uses the generative AI model to regenerate the broadcast content: "Today's news is that the stock market has risen significantly."

[0305] 5. The server adjusts the text and tone of the voice based on the emotional data.

[0306] 6. The server sends the final text and adjusted audio data to the device.

[0307] 7. The device displays the caption "Today's news: The stock market has risen significantly" on the screen and plays a voiceover in a calibrated surprised tone.

[0308] Prompt Sentence Examples

[0309] Below are some example prompts to input to a generative artificial intelligence model:

[0310] "Convert the user's speech, 'Show me the news subtitles,' into text and generate a natural-sounding sentence based on the speech. Also, convert the generated sentence into speech and adjust the tone and speed according to the user's level of surprise."

[0311] By combining these processes, the present invention makes it possible to realize a system that analyzes the voice input by the user, reproduces it as natural sentences, and provides a viewing experience that is in line with the user's emotions.

[0312] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0313] Step 1:

[0314] The user speaks into the microphone of the device to reproduce the audio of a television broadcast. For example, they say, "Show me the subtitles for the news." The input at this time is voice data.

[0315] Specifically, the user speaks into a microphone, which captures the speech.

[0316] Step 2:

[0317] The device captures the spoken voice in real time and transmits it to the server as digital voice data. At this time, the input is the voice acquired from the user in step 1, and the output is the digital voice data.

[0318] Specifically, the microphone captures the audio, converts the analog audio signal into a digital signal, and transmits it to a server via a network.

[0319] Step 3:

[0320] The server converts the received voice data into text data using speech recognition technology. The input here is digital voice data, and the output is text data. The technology used is a speech recognition engine (e.g., Google Cloud Speech-to-Text).

[0321] Specifically, the voice data is analyzed and converted into a corresponding character string.

[0322] Step 4:

[0323] The server uses a generative AI model (e.g., GPT-3) to regenerate the initial text data generated by speech recognition into natural, meaningful sentences. At this time, the input is the initial text data, and the output is the generated natural sentences.

[0324] Specifically, the initial text is input into a generative AI model, which then generates meaningful sentences taking into account context and grammar.

[0325] Step 5:

[0326] The server checks the grammar and punctuation of the generated text and fine-tunes it if necessary. The input is the regenerated text data, and the output is the final fine-tuned text data.

[0327] Specifically, the grammar check tool is used to detect and correct punctuation and grammatical errors.

[0328] Step 6:

[0329] The device analyzes the user's facial expression and tone of voice during voice input using an emotion recognition engine (e.g., Microsoft Azure Emotion API). The input is the user's facial expression and tone of voice, and the output is emotional data.

[0330] Specifically, it uses a camera and microphone to capture the user's facial expressions and tone of voice, and then analyzes the data in real time to recognize emotions.

[0331] Step 7:

[0332] The server adjusts the tone of the text and voice based on the emotion data. The input is the emotion data and the fine-tuned text data, and the output is the adjusted voice data and text data.

[0333] Specifically, it analyzes emotional data and appropriately adjusts the tone and speed of the generated text and audio based on the results.

[0334] Step 8:

[0335] The server converts the adjusted text data into audio data using text-to-speech technology (e.g., Amazon Polly), where the input is the adjusted text data and the output is audio data.

[0336] Specifically, the process involves using text-to-speech technology to convert text data into natural spoken voice.

[0337] Step 9:

[0338] The server transmits the generated voice data and the final text data to the terminal. The input here is the voice data and the text data, and the output is the data transmission to the terminal.

[0339] Specifically, the voice data and text data are transferred to the terminal via the network using a data transmission protocol.

[0340] Step 10:

[0341] The terminal displays the final text in scrolling format on the screen and plays the audio to the user through an audio output device (e.g., speaker or headphones). The input is the final text data and audio data, and the output is the provision of visual and auditory information to the user.

[0342] Specifically, the operation includes a screen control process for displaying text in a scrolling format and an audio playback process for controlling an audio playback device.

[0343] In this way, the system analyzes the user's speech, adjusts the generated text and audio to the user's emotions, and finally provides visual and audio information.

[0344] (Application example 2)

[0345] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0346] Conventional speech recognition systems can convert input speech into text and reproduce it, but they cannot provide a personalized viewing experience that reflects the user's emotions. Furthermore, in order to provide a comfortable interaction in real time, it is necessary to appropriately reflect emotional fluctuations. The present invention aims to solve these problems and provide a more comprehensive and personalized user experience.

[0347] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for converting input speech into text, means for regenerating the generated text into meaningful sentences, means for converting the regenerated text into speech and outputting it, and means for recognizing the user's emotion at the time of speech input and adjusting the tone of the generated text and speech based on that emotion. This enables dynamic text generation and speech output based on the user's emotion.

[0348] The "means for converting input speech into text" refers to a means for converting speech data into text data using speech recognition technology.

[0349] "Means for regenerating generated text into meaningful sentences" refers to means for using generative artificial intelligence to regenerate initial text data into natural-looking sentences that take into account context and grammar.

[0350] The "means for converting the regenerated text into speech and outputting it" refers to a means for converting the final text data into speech data using text-to-speech conversion technology and outputting it as speech to the user.

[0351] "Means for recognizing the user's emotions during voice input and adjusting the tone of the generated text and voice based on those emotions" refers to means for analyzing the user's facial expressions and tone of voice in real time and adjusting the tone of the text and voice based on the analysis results in accordance with the user's emotions.

[0352] This system converts speech input into text, regenerates it in a meaningful form, and finally outputs the regenerated text in a voice adjusted based on the user's emotions. This system consists of three elements: a server, a terminal, and a user.

[0353] System configuration

[0354] Collecting voice input and converting it to text

[0355] The user speaks into the microphone in the vehicle. The device captures this voice in real time and sends the voice data to the server. The server then converts the voice data into text data using voice recognition technology (e.g., Google Cloud Speech-to-Text).

[0356] Text analysis and generation

[0357] The server uses a generative artificial intelligence model (e.g., OpenAI's GPT-4) to regenerate the initial text data into meaningful sentences that take into account context and grammar, using prompts based on specific requests entered by the user.

[0358] An example prompt is:

[0359] Please make the user speak in natural sentences: Start navigation

[0360] Emotion collection and analysis

[0361] The device uses the vehicle's smart cameras and microphones to analyze the user's facial expressions and tone of voice in real time, and the analyzed emotional data is sent to a server, which then uses this data to adjust the tone and speed of the generated text and voice according to the user's emotions.

[0362] Text-to-speech conversion and output

[0363] The server converts the final text data into audio data using text-to-speech technology (e.g., Google Cloud Text-to-Speech). The tone and speed of the audio are adjusted based on the user's emotions. The adjusted audio and final text are then sent to the device, which outputs the audio through its speakers and displays the final text on the screen. This allows the user to enjoy a personalized listening experience that matches their emotions.

[0364] Specific examples

[0365] For example, consider a situation where a driver says, "Start navigation." If the system detects that the driver is tired, it generates the following sentence: "Start navigation to your destination. This road has little traffic." It then adjusts the tone of the generated sentence to be gentler. This adjusted voice is played through the vehicle's speakers, and the screen displays, "Start navigation to your destination. This road has little traffic."

[0366] This provides responses that are in line with the user's emotional state, making interactions more natural and engaging.

[0367] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0368] Step 1:

[0369] The device collects the user's voice input using a microphone. The input voice data is sent directly to the server. This allows the voice data to reach the server in real time.

[0370] Step 2:

[0371] The server converts the received voice data into text data using speech recognition technology (e.g., Google Cloud Speech-to-Text). The input here is voice data, and the output is the corresponding text data. Through this conversion process, the user's speech is represented in text format.

[0372] Step 3:

[0373] The server uses a generative artificial intelligence model (e.g., OpenAI's GPT-4) to regenerate the initial text data into meaningful sentences. The input is the initial text obtained by speech recognition technology, and the output is a regenerated natural sentence. Here, a prompt sentence is used to generate an appropriate response.

[0374] Step 4:

[0375] The terminal analyzes the user's facial expressions and tone of voice in real time using the smart camera and microphone in the vehicle. This collects the user's emotional data and sends it to the server. The input of this step is the user's facial and vocal data from the camera and microphone, and the output is emotional data.

[0376] Step 5:

[0377] The server adjusts the tone and speed of the generated text and voice based on the obtained emotion data according to the user's emotion. The input is emotion data and regenerated text data, and the output is text and voice data adjusted according to the emotion. This adjustment enables natural responses that are in line with the user's emotion.

[0378] Step 6:

[0379] The server converts the final text data into audio data using text-to-speech technology (e.g., Google Cloud Text-to-Speech), where the input is the adjusted text data and the output is the adjusted audio data.

[0380] Step 7:

[0381] The terminal plays the adjusted audio data sent from the server using an audio output device (speaker), and displays the final text data on the screen. The input is the adjusted audio data and text data, and the output is audio output and text display. This allows the user to have a personalized viewing experience that matches their emotions.

[0382] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0383] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0384] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0385] [Second embodiment]

[0386] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0387] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0388] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0389] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0390] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0391] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0392] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0393] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0394] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0395] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0396] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0397] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0398] A specific example of the configuration of a system for implementing the present invention is shown below.

[0399] System Overview

[0400] This system converts speech to text, regenerates the text, and converts the regenerated text back into speech for output, providing a comprehensive viewing experience. The specific processing flow and functions of this system are described below.

[0401] System configuration

[0402] This system consists of three elements: a server, a terminal, and a user. The system operates according to the following procedure.

[0403] Audio collection and text conversion

[0404] 1. Collecting voice input

[0405] The user inputs the audio of the television broadcast through the microphone of the device, for example, by issuing a voice command such as "display subtitles for the news" or by speaking the actual content of the television news broadcast.

[0406] The terminal captures this audio in real time and transmits the audio data as is to the server.

[0407] 2. Speech-to-text

[0408] The server converts the received voice data into text data using speech recognition technology. For example, a voice saying "Show me the subtitles for the news" is converted into text "Show me the subtitles for the news."

[0409] Text semantic generation and refinement

[0410] 3. Text Analysis and Generation

[0411] The server uses generative artificial intelligence to regenerate meaningful sentences based on the initial text generated by speech recognition technology, taking into account context and grammar to generate natural-sounding sentences. For example, the generated sentence might read, "In the news: The stock market rose significantly today."

[0412] 4. Text Tweaks

[0413] The server checks the generated text for grammar and punctuation and makes corrections if necessary. The final text is "In the news: The stock market rose significantly today."

[0414] Text-to-speech conversion and output

[0415] 5. Text to speech conversion

[0416] The server converts the final text into audio data using text-to-speech technology, and the resulting audio is played back as a natural voice.

[0417] 6. Text and audio transmission and output

[0418] The server transmits the generated voice data and the final text to the terminal.

[0419] The device displays subtitles on the screen, such as "In the news: The stock market saw a big jump today," while simultaneously playing audio data, allowing the user to consume both audio and text information.

[0420] Specific examples

[0421] For example, if a user says "Display subtitles for the news" in front of a TV, the system operates as follows.

[0422] 1. The user says, "Show me closed captions for the news."

[0423] 2. The device captures the audio and sends it to the server.

[0424] 3. The server uses speech recognition technology to convert the text into "Show me the news subtitles."

[0425] 4. The server uses generative artificial intelligence to recreate the broadcast: "In the news: The stock market rose significantly today."

[0426] 5. The server then converts this text into audio data using text-to-speech technology.

[0427] 6. The server sends the generated voice data and text to the terminal.

[0428] 7. The device will display the caption "In the news: The stock market saw a big jump today" on the screen and simultaneously play the audio.

[0429] In this way, the system converts speech to text, regenerates the text into meaningful sentences, and converts the regenerated text into speech for output, providing a comprehensive listening experience.

[0430] The processing flow will be explained below.

[0431] Step 1:

[0432] The user speaks the audio of the television broadcast into the device's microphone, for example, saying "display the news subtitles" or the actual content of the television news broadcast.

[0433] Step 2:

[0434] The terminal captures the user's voice in real time through a microphone and acquires it as voice data.

[0435] Step 3:

[0436] The device transmits the acquired voice data to a server via the Internet.

[0437] Step 4:

[0438] The server converts the received voice data into text data using voice recognition technology. For example, it converts the voice "Show me the subtitles for the news" into the text "Show me the subtitles for the news."

[0439] Step 5:

[0440] The server uses generative artificial intelligence to regenerate the initial text obtained from the speech recognition technology into a meaningful sentence based on the context, for example, "Today's news is that the stock market has risen significantly."

[0441] Step 6:

[0442] The server checks the generated text for grammar, punctuation, etc. and makes any necessary adjustments. The final text is "Today's news is that the stock market has risen significantly."

[0443] Step 7:

[0444] The server converts the final text data into audio data using text-to-speech technology, and the generated audio can be played back as a natural voice.

[0445] Step 8:

[0446] The server transmits the generated voice data and the final text to the terminal.

[0447] Step 9:

[0448] The terminal displays the final text "Today's news is that the stock market has risen significantly" in scrolling format on the screen.

[0449] Step 10:

[0450] The terminal reproduces the transmitted audio data to the user through an audio output device (speaker or headphones), allowing the user to use both audio information and text information.

[0451] Example 1

[0452] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0453] Conventional speech recognition systems simply convert speech to text without correcting the meaning or grammar of the text. This can result in unnatural and difficult-to-understand text for users. Furthermore, systems that convert the converted text back into speech and output it are limited. The present invention aims to solve these problems by providing a comprehensive system that converts speech to text, regenerates the text into natural-sounding sentences, and then re-outputs it as speech.

[0454] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0455] In this invention, the server includes means for converting input speech into text, means for regenerating the generated text into meaningful sentences, means for converting the regenerated text into speech and outputting it, means for capturing input speech in real time, means for checking the generated text with a grammar checker and correcting it if necessary, and means for providing the generated speech data and final text by screen display and audio playback, thereby enabling users to use natural and easy-to-understand text and speech information in real time.

[0456] "Speech recognition technology" is a technology that analyzes input voice data and converts it into text data.

[0457] "Generative AI" is an AI technology that generates natural-looking sentences based on input initial text, taking into account context and grammar.

[0458] "Text-to-speech technology" is a technology that converts text data into voice data and outputs it as natural voice.

[0459] "Capturing in real time" means that input audio data can be processed immediately as it is.

[0460] A "grammar checker" is a tool or technique for checking the grammar and punctuation of generated text and correcting errors.

[0461] "Displaying on screen" means displaying the generated text on a display device such as a monitor or screen.

[0462] "Audio playback" refers to playing back the generated audio data through an audio output device such as a speaker.

[0463] "Means" refers to a method, technique, or device used to achieve a particular purpose.

[0464] A "server" is a computer system connected to a network that processes requests from multiple terminals and provides data.

[0465] "Terminal" means a device that allows a user to input voice and display / play back the generated text and voice.

[0466] "User" refers to a person using the system to provide voice input.

[0467] A "network" is a communications infrastructure that allows multiple computer systems to communicate with each other.

[0468] An "input prompt" refers to the text of instructions or questions given to a generative artificial intelligence.

[0469] System Overview

[0470] The system of the present invention converts speech into text, regenerates the text into meaningful sentences, and finally converts the regenerated text into speech and outputs it. The system consists of three elements: a server, a terminal, and a user, and is realized using the following hardware and software.

[0471] Hardware and software used

[0472] 1. Hardware

[0473] Terminal: A device equipped with a microphone, display, and speaker that allows a user to input voice and output generated text and voice.

[0474] Server: Equipped with the computing resources to perform speech recognition, generative artificial intelligence, and text-to-speech conversion.

[0475] 2. Software

[0476] Speech recognition technology: Using Google Cloud Speech-to-Text, etc., voice data is converted into text data.

[0477] Generative artificial intelligence: Using models such as OpenAI GPT-4, initial text is generated into meaningful sentences.

[0478] Grammar checker: Checks and corrects the grammar of the generated text using APIs such as Grammarly.

[0479] Text-to-speech technology: Using technologies such as Amazon Polly, text data is converted into natural-sounding speech data.

[0480] System Operation

[0481] 1. The user stands in front of the device and gives a voice command, such as "Show subtitles for the news." The device's microphone captures this voice in real time and sends the voice data to the server.

[0482] 2. The server converts the received voice data into text using Google Cloud Speech-to-Text. At this time, the voice command "Show news subtitles" is converted into text data "Show news subtitles."

[0483] 3. The server uses generative artificial intelligence (OpenAI GPT-4) to regenerate the text data into natural-sounding sentences. For example, if a user is watching the news on TV, the initial text "Show me the news subtitles" is generated into a detailed sentence: "News: The stock market rose significantly today." The input prompt for the generative AI model will be in the following text format:

[0484] Example prompt: "Regenerate this text into a meaningful sentence. Original text: 'In the news: The stock market rose significantly today.'"

[0485] 4. The server checks the generated text with a grammar checker (Grammarly API) and corrects any errors or unnatural points to generate the final text.

[0486] 5. The server converts the final text into speech using Amazon Polly, generating a natural-sounding speech, for example, "In the news: The stock market rose significantly today."

[0487] 6. The server sends the generated voice data and the final text to the terminal, which displays "In the news: The stock market rose significantly today" on the screen and plays the voice at the same time.

[0488] Specific examples

[0489] As a specific example, the system operation when the user utters "Display subtitles for the news" will be described.

[0490] 1. The user says, "Show me closed captions for the news."

[0491] 2. The device captures the audio and sends it to the server.

[0492] 3. The server converts the audio data into text.

[0493] 4. The server uses the generative AI model to regenerate the initial text into detailed sentences.

[0494] 5. The server checks the generated text with a grammar checker and corrects it.

[0495] 6. The server converts the final text into audio data.

[0496] 7. The server sends the voice data and text to the device.

[0497] 8. The device displays the final text and plays the audio.

[0498] In this way, the system of the present invention can convert speech into text and speech in real time, providing information to the user in a natural and easy-to-understand manner.

[0499] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0500] Step 1:

[0501] The process begins when a user speaks a command in front of the device. For example, they might say, "Display subtitles for the news." This voice is captured by the device's microphone and saved as audio data in real time. The input is audio data, and the output is the audio data saved on the device. The device then prepares to send this audio data to the server via the network.

[0502] Step 2:

[0503] The device transmits the captured audio data to the server via the network. The input here is the audio data stored on the device, and the output is the audio data to be transmitted to the server. The device establishes a network connection and executes communication processing to transfer the audio data to the server.

[0504] Step 3:

[0505] The server converts the voice data received from the device into text data using voice recognition technology such as Google Cloud Speech-to-Text. The input is the voice data received by the server, and the output is text data. The server analyzes the voice data and performs the specific operation of calling a voice recognition API to convert it into text.

[0506] Step 4:

[0507] The server uses generative artificial intelligence (e.g., OpenAI GPT-4) to regenerate the initial text into a meaningful sentence. The input is text data obtained by speech recognition technology, and the output is the regenerated sentence. The specific format of the prompt sentence is, "Please regenerate this text into a meaningful sentence. Original text: 'News: The stock market rose significantly today.'" The server sends the prompt sentence to the generative AI model and obtains the generated sentence.

[0508] Step 5:

[0509] The server checks the generated text with a grammar checker (e.g., Grammarly API) and corrects grammar and punctuation. The input is the text generated by the generative AI model, and the output is the final text after corrections. The server sends the text to the grammar checker, receives correction feedback, and incorporates it.

[0510] Step 6:

[0511] The server converts the final corrected text into audio data using text-to-speech technology such as Amazon Polly. The input is the final text and the output is audio data. The server calls the text-to-speech API to convert the text into audio and obtains the generated audio data.

[0512] Step 7:

[0513] The server sends the generated voice data and final text to the terminal. The input is the generated voice data and final text, and the output is the data to be sent to the terminal. The server performs communication processing to transfer the voice data and text to the terminal via the network.

[0514] Step 8:

[0515] The terminal displays the final text received from the server on the screen and simultaneously plays the audio data from the speaker. The input is the audio data and text data received from the server, and the output is the text display on the screen and audio playback from the speaker. Specifically, the terminal outputs the final text to the display and plays the audio data on the audio device.

[0516] By performing the above steps, the system of the present invention can convert speech into text and speech in real time, and provide information to the user in a natural and easy-to-understand format.

[0517] (Application example 1)

[0518] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0519] Conventional content viewing systems lack a mechanism for providing real-time subtitles or summaries of the content users are watching, which results in a lack of an improved viewing experience for users, making it difficult to understand specific scenes or easily check the introduction of the next episode through audio and text.

[0520] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0521] In this invention, the server includes means for converting input speech into text, means for regenerating the generated text into meaningful sentences, means for converting the regenerated text into speech and outputting it, means for displaying the generated text on a display device, means for providing the output speech using a speech output device, means for summarizing the next content based on the generated text, and communication means for transmitting the generated text and speech data to a user terminal, thereby enabling the user to check subtitles and content summaries in both speech and text in real time.

[0522] The "means for converting input voice into text" refers to a means for analyzing voice data input by a user and converting the content of the voice data into text data of the corresponding sentence.

[0523] The "means for regenerating generated text into meaningful sentences" refers to a means for processing the text generated by the initial speech recognition to reconstruct it into natural sentences based on context and grammar.

[0524] The "means for converting the regenerated text into speech and outputting the speech" refers to a means for converting the regenerated text data into speech data and performing processing to output the speech data to the user.

[0525] "Means for displaying the generated text on a display device" refers to means for transmitting the generated text data to a display device such as a display or screen in order to visually display it, and displaying it in a form that can be confirmed by the user.

[0526] The "means for providing audio output using an audio output device" refers to a means for allowing a user to listen to the generated audio data via an audio output device such as a speaker or headphones.

[0527] The "means for summarizing the next content based on the generated text" is a processing means for briefly summarizing the next episode or important content based on the initial input speech and the regenerated text.

[0528] "Communication means for transmitting the generated text and voice data to the user's terminal" refers to a means including communication technology for transmitting the generated text and voice data from the server to a terminal such as the user's smartphone or head-mounted display.

[0529] System Overview

[0530] This invention is a system that provides subtitles and summaries of content in real time in response to user voice commands while viewing content. This system consists of three main elements: a server, a terminal, and a user, and these elements work in conjunction with each other.

[0531] Hardware and Software Configuration

[0532] server:

[0533] Speech recognition engine used: Google Speech-to-Text API

[0534] Generative artificial intelligence model used: OpenAI GPT model

[0535] Text-to-speech engine used: Amazon Polly

[0536] Communication method: Internet communication

[0537] Device:

[0538] Audio input device: Microphone (built into smartphone or head-mounted display)

[0539] Audio output device: Speakers or headphones

[0540] Display devices: smartphone and head-mounted display screens

[0541] Processing flow

[0542] When a user uses a device to give voice commands, the device's microphone captures the voice and sends it to a server over the Internet. The server converts the voice into text using a speech recognition engine (Google Speech-to-Text API). It then uses a generative artificial intelligence model (OpenAI GPT model) to regenerate the text into natural-sounding sentences appropriate to the context. The generated text is then converted into audio data by a text-to-speech engine (Amazon Polly) and sent back to the device over the Internet. Finally, the device displays the text on the screen and plays the audio through speakers or headphones.

[0543] Specific example explanation

[0544] Example 1: Displaying subtitles

[0545] While watching a movie, a user says, "Show subtitles." The device's microphone captures the audio and sends it to the server. The server converts the audio to text, regenerates it, converts it back to audio, and sends it to the device. The device displays on the screen, "Movie content: The protagonist sets off on a new adventure." and simultaneously plays the audio.

[0546] Example 2: Next episode introduction

[0547] While watching a drama, a user says, "Please tell me about the next episode." The device's microphone captures the audio and sends it to the server. The server converts the audio into text and regenerates it based on the context. The generated text reads, "In the next episode, the protagonist sets out on a new adventure." This is converted into audio and sent to the device. The device displays "In the next episode, the protagonist sets out on a new adventure" on the screen and simultaneously plays it aloud.

[0548] Prompt Sentence Examples

[0549] User voice: "Please introduce the next episode."

[0550] Translated text: "Please introduce the next episode"

[0551] Example of a generative AI model input prompt: "Please explain the content of the next episode."

[0552] Generated output: "In the next episode, our protagonist sets off on a new adventure, facing various obstacles and meeting new allies."

[0553] In this way, the system converts speech to text and displays and speaks the reproduced text, providing a user with an enhanced viewing experience.

[0554] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0555] Step 1:

[0556] The user gives a voice command. For example, the user says, "Please introduce the next episode." This becomes the voice input data.

[0557] Step 2:

[0558] The device's microphone captures the voice input data, which is then sent by the device to the server.

[0559] Step 3:

[0560] The server receives the voice data and converts it into text data using a speech recognition engine (Google Speech-to-Text API). The input in this step is voice data, and the output is text data.

[0561] Step 4:

[0562] The server uses a generative artificial intelligence model (OpenAI GPT model) to regenerate the converted text data into meaningful sentences. In this step, it takes the text data as input and generates natural-looking sentences that are appropriate for the context. The output is the regenerated text.

[0563] Step 5:

[0564] The server converts the regenerated text data into audio data using a text-to-speech engine (Amazon Polly). The input is the regenerated text, and the output is audio data.

[0565] Step 6:

[0566] The server sends the generated voice data and text data to the terminal, which outputs the voice data and text data.

[0567] Step 7:

[0568] The text data received by the terminal is displayed on the screen. At the same time, the audio data is played back using the terminal's audio output device (speaker or headphones). This allows the user to see the text on the screen and hear the content aloud. The input is text data and audio data, and the output is screen display and audio playback.

[0569] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0570] A specific example of the configuration of a system for implementing the present invention is shown below.

[0571] System Overview

[0572] This system not only converts speech to text, regenerates the text, and converts the regenerated text back into speech for output, but also recognizes the user's emotions and reflects them in the output content, providing a more comprehensive and personalized viewing experience. The specific processing flow and functions of this system are described below.

[0573] System configuration

[0574] This system consists of three elements: a server, a terminal, and a user. The system operates according to the following procedure.

[0575] Audio collection and text conversion

[0576] 1. Collecting voice input

[0577] The user speaks the audio of the television broadcast into the microphone of the terminal, for example, "display the subtitles of the news" or the actual content of the television news broadcast.

[0578] The terminal captures this audio in real time and transmits the audio data as is to the server.

[0579] 2. Speech-to-text

[0580] The server converts the received voice data into text data using voice recognition technology. For example, it converts the voice "Show me the subtitles for the news" into the text "Show me the subtitles for the news."

[0581] Text semantic generation and refinement

[0582] 3. Text Analysis and Generation

[0583] The server uses generative artificial intelligence to regenerate meaningful sentences based on the initial text generated by speech recognition technology. At this time, the server takes into account context and grammar to generate natural-sounding sentences. For example, it generates a sentence like, "Today's news is that the stock market has risen significantly."

[0584] 4. Text Tweaks

[0585] The server checks the generated text for grammar, punctuation, etc. and makes any necessary adjustments. The final text is "Today's news is that the stock market has risen significantly."

[0586] User Emotion Recognition

[0587] 5. Emotion collection and analysis

[0588] The device uses an emotion recognition engine to analyze the user's facial expressions and tone of voice during voice input. This analysis is performed in real time and emotion data is generated.

[0589] The server receives the emotion data and reflects it in the generated text and voice.

[0590] Text-to-speech conversion and output

[0591] 6. Text to speech conversion

[0592] The server then converts the final text data into audio data using text-to-speech technology, with the tone and speed of the generated audio adjusted to match the user's emotions.

[0593] 7. Text and audio transmission and output

[0594] The server transmits the generated voice data and the final text to the terminal.

[0595] The terminal displays the final text "Today's news is that the stock market has risen significantly" in scrolling format on the screen.

[0596] The device then plays the audio data back to the user through an audio output device (speaker or headphones). The audio is tailored to the user's emotions, providing a more personalized listening experience.

[0597] Specific examples

[0598] For example, if a user stands in front of a TV and says, "Display subtitles for the news," with a surprised expression on their face, the system operates as follows.

[0599] 1. The user looks surprised while saying, "Show me the news subtitles."

[0600] 2. The device captures the voice and facial expressions and sends the voice and emotion data to the server.

[0601] 3. The server uses speech recognition technology to convert the text into "Show me the news subtitles."

[0602] 4. The server uses generative artificial intelligence to recreate the broadcast: "In the news: The stock market rose significantly today."

[0603] 5. The server adjusts the text and tone of the voice based on the emotional data.

[0604] 6. The server sends the final text and adjusted audio data to the device.

[0605] 7. The device displays the caption "In the news: The stock market rose significantly today" on the screen and plays a voiceover in a calibrated tone of surprise.

[0606] In this way, the system converts speech to text and then converts the regenerated text into speech that matches the user's emotions, providing a personalized listening experience.

[0607] The processing flow will be explained below.

[0608] Step 1:

[0609] The user speaks the audio of the television broadcast into the device's microphone, for example, saying "display the news subtitles" or the actual content of the television news broadcast.

[0610] Step 2:

[0611] The device captures the user's voice in real time through a microphone and acquires it as voice data, while also collecting emotional information such as the user's facial expressions and tone of voice.

[0612] Step 3:

[0613] The device transmits the acquired voice data and emotion data to a server via the Internet.

[0614] Step 4:

[0615] The server converts the received voice data into text data using voice recognition technology. For example, it converts the voice "Show me the subtitles for the news" into the text "Show me the subtitles for the news."

[0616] Step 5:

[0617] The server uses generative artificial intelligence to regenerate meaningful sentences based on the initial text generated by speech recognition technology. At this time, it takes into account context and grammar to generate natural sentences. For example, the generated sentence might be, "Today's news is that the stock market has risen significantly."

[0618] Step 6:

[0619] The server checks the generated text for grammar, punctuation, etc. and makes any necessary adjustments. The final text is "Today's news is that the stock market has risen significantly."

[0620] Step 7:

[0621] The server analyzes the received emotional data to recognize the user's emotional state, which includes facial expressions such as joy, surprise, and sadness, as well as tone of voice.

[0622] Step 8:

[0623] The server adjusts the text and audio output depending on the user's emotion. For example, if the emotion is "surprise," it will emphasize the text, raise the tone of the audio, and increase the playback speed.

[0624] Step 9:

[0625] The server converts the final text into audio data using text-to-speech technology, and the generated audio is adjusted in tone and speed to match the user's emotions.

[0626] Step 10:

[0627] The server sends the adjusted voice data and the final text to the terminal.

[0628] Step 11:

[0629] The terminal displays the final text "Today's news is that the stock market has risen significantly" in scrolling format on the screen.

[0630] Step 12:

[0631] The device then plays the audio data back to the user through an audio output device (speaker or headphones). The audio is tailored to the user's emotions, providing a more personalized listening experience.

[0632] Example 2

[0633] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0634] Conventional speech recognition systems often convert input speech into text and output the text as is. This makes it difficult to provide a personalized listening experience that reflects the user's emotions. Furthermore, the generated text often lacks proper grammar and tone adjustment, which can detract from a natural conversational experience. Furthermore, the visual display and audio output are not integrated, preventing comprehensive information provision.

[0635] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0636] In this invention, the server includes means for converting input speech into text, means for regenerating the generated text into meaningful sentences, means for recognizing a user's emotion and adjusting the tone and speed of the regenerated text and speech based on the emotion, means for converting the regenerated text into speech and outputting it, means for displaying the final text in a scrolling format on a display device, and means for providing the output speech to the user using an audio output device, thereby enabling a natural conversation experience and a personalized listening and listening experience that is tailored to the user's emotion.

[0637] The "means for converting input speech into text" refers to a system or technology that analyzes speech data input by a user and converts it into corresponding text data.

[0638] A "means for regenerating generated text into meaningful sentences" is a technology or system that uses initial text data to reconstruct sentences that have natural meaning, taking into account context and grammar.

[0639] "Means for recognizing user emotions" refers to technology or a system that analyzes the user's facial expressions and tone of voice to understand their emotional state at that time.

[0640] The "means for converting the reproduced text into speech and outputting it" refers to a technology or system that analyzes text data, converts it into corresponding speech data, and plays it back to the user.

[0641] The "means for displaying in a scrolling format on a display device" refers to a technology or system for visually displaying text data in a format that moves continuously on the display device.

[0642] "Speech recognition technology" is a technology that analyzes a voice signal and generates corresponding text data.

[0643] A "generative AI model" is a model that uses artificial intelligence to generate meaningful sentences from given text data.

[0644] An "emotion recognition engine" is a technology or system that analyzes a user's facial expressions and tone of voice to identify their emotional state.

[0645] "Text-to-speech technology" is a technology that analyzes text data and generates speech data based on its content.

[0646] An "audio output device" is a device or system that reproduces audio data and allows the user to listen to it.

[0647] The system of the present invention converts voice input into text, then regenerates the text into meaningful sentences, and further recognizes the user's emotions to adjust the tone and speed of the generated text and speech, providing a more comprehensive and personalized listening experience. This system consists of three elements: a server, a terminal, and a user, and operates according to the following process flow.

[0648] Audio collection and text conversion

[0649] The user speaks the audio of the television broadcast into the microphone of the terminal, for example, saying "Display subtitles for the news."

[0650] The device captures this voice in real time and generates digital audio data, which is then immediately sent to the server.

[0651] The server converts the received voice data into text data using speech recognition technology (commonly known as a speech recognition engine). A major speech recognition technology is Google Cloud Speech-to-Text. The converted text data is used in the next step.

[0652] Text semantic generation and refinement

[0653] The server uses a generative AI model (commonly known as a language generation model) to regenerate meaningful sentences based on the initial text generated by speech recognition. This series of processes uses GPT-3 and other technologies. Specifically, based on the text obtained through speech recognition, "Show me the subtitles for the news," it generates a natural-sounding sentence that is in line with the context, "Today's news is that the stock market has risen significantly."

[0654] After the text is generated, the server checks it for grammar and punctuation and makes any necessary adjustments, resulting in the final text sounding natural: "Today's news is that the stock market has risen significantly."

[0655] User Emotion Recognition

[0656] The device uses an emotion recognition engine (commonly known as an emotion analysis system) to analyze the user's facial expressions and tone of voice when they input voice. This analysis is performed in real time, and user emotion data is generated. A major emotion recognition engine is the Microsoft Azure Emotion API.

[0657] The server receives the emotion data and reflects it in the generated text and voice. Specifically, it adjusts the tone of the text or speeds up the voice depending on the emotion of surprise.

[0658] Text-to-speech conversion and output

[0659] The server then converts the final text data into audio data using text-to-speech technology (commonly known as a text-to-speech engine). Amazon Polly is used for this conversion. The generated audio is adjusted in tone and speed to match the user's emotion. For example, if surprise is recognized, a voice will be generated saying, "Today's news is that the stock market has risen significantly," in a surprised tone.

[0660] The server sends the generated voice data and the final text to the terminal. The terminal displays the final text, "Today's news is that the stock market has risen significantly," in a scrolling format on the screen. The terminal also plays the sent voice data through an audio output device (e.g., speaker or headphones). This allows the news to be conveyed in a tone that corresponds to the emotion.

[0661] Specific example explanation

[0662] The operation when a user is in front of a television and says "Display subtitles for the news" while making a surprised expression is as follows.

[0663] 1. The user looks surprised while saying, "Show me the news subtitles."

[0664] 2. The device captures the voice and facial expressions and sends the voice and emotion data to the server.

[0665] 3. The server uses speech recognition technology to convert the audio "Show me the news subtitles" into the text "Show me the news subtitles."

[0666] 4. The server uses the generative AI model to regenerate the broadcast content: "Today's news is that the stock market has risen significantly."

[0667] 5. The server adjusts the text and tone of the voice based on the emotional data.

[0668] 6. The server sends the final text and adjusted audio data to the device.

[0669] 7. The device displays the caption "Today's news: The stock market has risen significantly" on the screen and plays a voiceover in a calibrated surprised tone.

[0670] Prompt Sentence Examples

[0671] Below are some example prompts to input to a generative artificial intelligence model:

[0672] "Convert the user's speech, 'Show me the news subtitles,' into text and generate a natural-sounding sentence based on the speech. Also, convert the generated sentence into speech and adjust the tone and speed according to the user's level of surprise."

[0673] By combining these processes, the present invention makes it possible to realize a system that analyzes the voice input by the user, reproduces it as natural sentences, and provides a viewing experience that is in line with the user's emotions.

[0674] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0675] Step 1:

[0676] The user speaks into the microphone of the device to reproduce the audio of a television broadcast. For example, they say, "Show me the subtitles for the news." The input at this time is voice data.

[0677] Specifically, the user speaks into a microphone, which captures the speech.

[0678] Step 2:

[0679] The device captures the spoken voice in real time and transmits it to the server as digital voice data. At this time, the input is the voice acquired from the user in step 1, and the output is the digital voice data.

[0680] Specifically, the microphone captures the audio, converts the analog audio signal into a digital signal, and transmits it to a server via a network.

[0681] Step 3:

[0682] The server converts the received voice data into text data using speech recognition technology. The input here is digital voice data, and the output is text data. The technology used is a speech recognition engine (e.g., Google Cloud Speech-to-Text).

[0683] Specifically, the voice data is analyzed and converted into a corresponding character string.

[0684] Step 4:

[0685] The server uses a generative AI model (e.g., GPT-3) to regenerate the initial text data generated by speech recognition into natural, meaningful sentences. At this time, the input is the initial text data, and the output is the generated natural sentences.

[0686] Specifically, the initial text is input into a generative AI model, which then generates meaningful sentences taking into account context and grammar.

[0687] Step 5:

[0688] The server checks the grammar and punctuation of the generated text and fine-tunes it if necessary. The input is the regenerated text data, and the output is the final fine-tuned text data.

[0689] Specifically, the grammar check tool is used to detect and correct punctuation and grammatical errors.

[0690] Step 6:

[0691] The device analyzes the user's facial expression and tone of voice during voice input using an emotion recognition engine (e.g., Microsoft Azure Emotion API). The input is the user's facial expression and tone of voice, and the output is emotional data.

[0692] Specifically, it uses a camera and microphone to capture the user's facial expressions and tone of voice, and then analyzes the data in real time to recognize emotions.

[0693] Step 7:

[0694] The server adjusts the tone of the text and voice based on the emotion data. The input is the emotion data and the fine-tuned text data, and the output is the adjusted voice data and text data.

[0695] Specifically, it analyzes emotional data and appropriately adjusts the tone and speed of the generated text and audio based on the results.

[0696] Step 8:

[0697] The server converts the adjusted text data into audio data using text-to-speech technology (e.g., Amazon Polly), where the input is the adjusted text data and the output is audio data.

[0698] Specifically, the process involves using text-to-speech technology to convert text data into natural spoken voice.

[0699] Step 9:

[0700] The server transmits the generated voice data and the final text data to the terminal. The input here is the voice data and the text data, and the output is the data transmission to the terminal.

[0701] Specifically, the voice data and text data are transferred to the terminal via the network using a data transmission protocol.

[0702] Step 10:

[0703] The terminal displays the final text in scrolling format on the screen and plays the audio to the user through an audio output device (e.g., speaker or headphones). The input is the final text data and audio data, and the output is the provision of visual and auditory information to the user.

[0704] Specifically, the operation includes a screen control process for displaying text in a scrolling format and an audio playback process for controlling an audio playback device.

[0705] In this way, the system analyzes the user's speech, adjusts the generated text and audio to the user's emotions, and finally provides visual and audio information.

[0706] (Application example 2)

[0707] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0708] Conventional speech recognition systems can convert input speech into text and reproduce it, but they cannot provide a personalized viewing experience that reflects the user's emotions. Furthermore, in order to provide a comfortable interaction in real time, it is necessary to appropriately reflect emotional fluctuations. The present invention aims to solve these problems and provide a more comprehensive and personalized user experience.

[0709] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for converting input speech into text, means for regenerating the generated text into meaningful sentences, means for converting the regenerated text into speech and outputting it, and means for recognizing the user's emotion at the time of speech input and adjusting the tone of the generated text and speech based on that emotion. This enables dynamic text generation and speech output based on the user's emotion.

[0710] The "means for converting input speech into text" refers to a means for converting speech data into text data using speech recognition technology.

[0711] "Means for regenerating generated text into meaningful sentences" refers to means for using generative artificial intelligence to regenerate initial text data into natural-looking sentences that take into account context and grammar.

[0712] The "means for converting the regenerated text into speech and outputting it" refers to a means for converting the final text data into speech data using text-to-speech conversion technology and outputting it as speech to the user.

[0713] "Means for recognizing the user's emotions during voice input and adjusting the tone of the generated text and voice based on those emotions" refers to means for analyzing the user's facial expressions and tone of voice in real time and adjusting the tone of the text and voice based on the analysis results in accordance with the user's emotions.

[0714] This system converts speech input into text, regenerates it in a meaningful form, and finally outputs the regenerated text in a voice adjusted based on the user's emotions. This system consists of three elements: a server, a terminal, and a user.

[0715] System configuration

[0716] Collecting voice input and converting it to text

[0717] The user speaks into the microphone in the vehicle. The device captures this voice in real time and sends the voice data to the server. The server then converts the voice data into text data using voice recognition technology (e.g., Google Cloud Speech-to-Text).

[0718] Text analysis and generation

[0719] The server uses a generative artificial intelligence model (e.g., OpenAI's GPT-4) to regenerate the initial text data into meaningful sentences that take into account context and grammar, using prompts based on specific requests entered by the user.

[0720] An example prompt is:

[0721] Please make the user speak in natural sentences: Start navigation

[0722] Emotion collection and analysis

[0723] The device uses the vehicle's smart cameras and microphones to analyze the user's facial expressions and tone of voice in real time, and the analyzed emotional data is sent to a server, which then uses this data to adjust the tone and speed of the generated text and voice according to the user's emotions.

[0724] Text-to-speech conversion and output

[0725] The server converts the final text data into audio data using text-to-speech technology (e.g., Google Cloud Text-to-Speech). The tone and speed of the audio are adjusted based on the user's emotions. The adjusted audio and final text are then sent to the device, which outputs the audio through its speakers and displays the final text on the screen. This allows the user to enjoy a personalized listening experience that matches their emotions.

[0726] Specific examples

[0727] For example, consider a situation where a driver says, "Start navigation." If the system detects that the driver is tired, it generates the following sentence: "Start navigation to your destination. This road has little traffic." It then adjusts the tone of the generated sentence to be gentler. This adjusted voice is played through the vehicle's speakers, and the screen displays, "Start navigation to your destination. This road has little traffic."

[0728] This provides responses that are in line with the user's emotional state, making interactions more natural and engaging.

[0729] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0730] Step 1:

[0731] The device collects the user's voice input using a microphone. The input voice data is sent directly to the server. This allows the voice data to reach the server in real time.

[0732] Step 2:

[0733] The server converts the received voice data into text data using speech recognition technology (e.g., Google Cloud Speech-to-Text). The input here is voice data, and the output is the corresponding text data. Through this conversion process, the user's speech is represented in text format.

[0734] Step 3:

[0735] The server uses a generative artificial intelligence model (e.g., OpenAI's GPT-4) to regenerate the initial text data into meaningful sentences. The input is the initial text obtained by speech recognition technology, and the output is a regenerated natural sentence. Here, a prompt sentence is used to generate an appropriate response.

[0736] Step 4:

[0737] The terminal analyzes the user's facial expressions and tone of voice in real time using the smart camera and microphone in the vehicle. This collects the user's emotional data and sends it to the server. The input of this step is the user's facial and vocal data from the camera and microphone, and the output is emotional data.

[0738] Step 5:

[0739] The server adjusts the tone and speed of the generated text and voice based on the obtained emotion data according to the user's emotion. The input is emotion data and regenerated text data, and the output is text and voice data adjusted according to the emotion. This adjustment enables natural responses that are in line with the user's emotion.

[0740] Step 6:

[0741] The server converts the final text data into audio data using text-to-speech technology (e.g., Google Cloud Text-to-Speech), where the input is the adjusted text data and the output is the adjusted audio data.

[0742] Step 7:

[0743] The terminal plays the adjusted audio data sent from the server using an audio output device (speaker), and displays the final text data on the screen. The input is the adjusted audio data and text data, and the output is audio output and text display. This allows the user to have a personalized viewing experience that matches their emotions.

[0744] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0745] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0746] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0747] [Third embodiment]

[0748] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0749] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0750] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0751] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0752] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0753] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0754] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0755] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0756] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0757] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0758] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0759] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0760] A specific example of the configuration of a system for implementing the present invention is shown below.

[0761] System Overview

[0762] This system converts speech to text, regenerates the text, and converts the regenerated text back into speech for output, providing a comprehensive viewing experience. The specific processing flow and functions of this system are described below.

[0763] System configuration

[0764] This system consists of three elements: a server, a terminal, and a user. The system operates according to the following procedure.

[0765] Audio collection and text conversion

[0766] 1. Collecting voice input

[0767] The user inputs the audio of the television broadcast through the microphone of the device, for example, by issuing a voice command such as "display subtitles for the news" or by speaking the actual content of the television news broadcast.

[0768] The terminal captures this audio in real time and transmits the audio data as is to the server.

[0769] 2. Speech-to-text

[0770] The server converts the received voice data into text data using speech recognition technology. For example, a voice saying "Show me the subtitles for the news" is converted into text "Show me the subtitles for the news."

[0771] Text semantic generation and refinement

[0772] 3. Text Analysis and Generation

[0773] The server uses generative artificial intelligence to regenerate meaningful sentences based on the initial text generated by speech recognition technology, taking into account context and grammar to generate natural-sounding sentences. For example, the generated sentence might read, "In the news: The stock market rose significantly today."

[0774] 4. Text Tweaks

[0775] The server checks the generated text for grammar and punctuation and makes corrections if necessary. The final text is "In the news: The stock market rose significantly today."

[0776] Text-to-speech conversion and output

[0777] 5. Text to speech conversion

[0778] The server converts the final text into audio data using text-to-speech technology, and the resulting audio is played back as a natural voice.

[0779] 6. Text and audio transmission and output

[0780] The server transmits the generated voice data and the final text to the terminal.

[0781] The device displays subtitles on the screen, such as "In the news: The stock market saw a big jump today," while simultaneously playing audio data, allowing the user to consume both audio and text information.

[0782] Specific examples

[0783] For example, if a user says "Display subtitles for the news" in front of a TV, the system operates as follows.

[0784] 1. The user says, "Show me closed captions for the news."

[0785] 2. The device captures the audio and sends it to the server.

[0786] 3. The server uses speech recognition technology to convert the text into "Show me the news subtitles."

[0787] 4. The server uses generative artificial intelligence to recreate the broadcast: "In the news: The stock market rose significantly today."

[0788] 5. The server then converts this text into audio data using text-to-speech technology.

[0789] 6. The server sends the generated voice data and text to the terminal.

[0790] 7. The device will display the caption "In the news: The stock market saw a big jump today" on the screen and simultaneously play the audio.

[0791] In this way, the system converts speech to text, regenerates the text into meaningful sentences, and converts the regenerated text into speech for output, providing a comprehensive listening experience.

[0792] The processing flow will be explained below.

[0793] Step 1:

[0794] The user speaks the audio of the television broadcast into the device's microphone, for example, saying "display the news subtitles" or the actual content of the television news broadcast.

[0795] Step 2:

[0796] The terminal captures the user's voice in real time through a microphone and acquires it as voice data.

[0797] Step 3:

[0798] The device transmits the acquired voice data to a server via the Internet.

[0799] Step 4:

[0800] The server converts the received voice data into text data using voice recognition technology. For example, it converts the voice "Show me the subtitles for the news" into the text "Show me the subtitles for the news."

[0801] Step 5:

[0802] The server uses generative artificial intelligence to regenerate the initial text obtained from the speech recognition technology into a meaningful sentence based on the context, for example, "Today's news is that the stock market has risen significantly."

[0803] Step 6:

[0804] The server checks the generated text for grammar, punctuation, etc. and makes any necessary adjustments. The final text is "Today's news is that the stock market has risen significantly."

[0805] Step 7:

[0806] The server converts the final text data into audio data using text-to-speech technology, and the generated audio can be played back as a natural voice.

[0807] Step 8:

[0808] The server transmits the generated voice data and the final text to the terminal.

[0809] Step 9:

[0810] The terminal displays the final text "Today's news is that the stock market has risen significantly" in scrolling format on the screen.

[0811] Step 10:

[0812] The terminal reproduces the transmitted audio data to the user through an audio output device (speaker or headphones), allowing the user to use both audio information and text information.

[0813] Example 1

[0814] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0815] Conventional speech recognition systems simply convert speech to text without correcting the meaning or grammar of the text. This can result in unnatural and difficult-to-understand text for users. Furthermore, systems that convert the converted text back into speech and output it are limited. The present invention aims to solve these problems by providing a comprehensive system that converts speech to text, regenerates the text into natural-sounding sentences, and then re-outputs it as speech.

[0816] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0817] In this invention, the server includes means for converting input speech into text, means for regenerating the generated text into meaningful sentences, means for converting the regenerated text into speech and outputting it, means for capturing input speech in real time, means for checking the generated text with a grammar checker and correcting it if necessary, and means for providing the generated speech data and final text by screen display and audio playback, thereby enabling users to use natural and easy-to-understand text and speech information in real time.

[0818] "Speech recognition technology" is a technology that analyzes input voice data and converts it into text data.

[0819] "Generative AI" is an AI technology that generates natural-looking sentences based on input initial text, taking into account context and grammar.

[0820] "Text-to-speech technology" is a technology that converts text data into voice data and outputs it as natural voice.

[0821] "Capturing in real time" means that input audio data can be processed immediately as it is.

[0822] A "grammar checker" is a tool or technique for checking the grammar and punctuation of generated text and correcting errors.

[0823] "Displaying on screen" means displaying the generated text on a display device such as a monitor or screen.

[0824] "Audio playback" refers to playing back the generated audio data through an audio output device such as a speaker.

[0825] "Means" refers to a method, technique, or device used to achieve a particular purpose.

[0826] A "server" is a computer system connected to a network that processes requests from multiple terminals and provides data.

[0827] "Terminal" means a device that allows a user to input voice and display / play back the generated text and voice.

[0828] "User" refers to a person using the system to provide voice input.

[0829] A "network" is a communications infrastructure that allows multiple computer systems to communicate with each other.

[0830] An "input prompt" refers to the text of instructions or questions given to a generative artificial intelligence.

[0831] System Overview

[0832] The system of the present invention converts speech into text, regenerates the text into meaningful sentences, and finally converts the regenerated text into speech and outputs it. The system consists of three elements: a server, a terminal, and a user, and is realized using the following hardware and software.

[0833] Hardware and software used

[0834] 1. Hardware

[0835] Terminal: A device equipped with a microphone, display, and speaker that allows a user to input voice and output generated text and voice.

[0836] Server: Equipped with the computing resources to perform speech recognition, generative artificial intelligence, and text-to-speech conversion.

[0837] 2. Software

[0838] Speech recognition technology: Using Google Cloud Speech-to-Text, etc., voice data is converted into text data.

[0839] Generative artificial intelligence: Using models such as OpenAI GPT-4, initial text is generated into meaningful sentences.

[0840] Grammar checker: Checks and corrects the grammar of the generated text using APIs such as Grammarly.

[0841] Text-to-speech technology: Using technologies such as Amazon Polly, text data is converted into natural-sounding speech data.

[0842] System Operation

[0843] 1. The user stands in front of the device and gives a voice command, such as "Show subtitles for the news." The device's microphone captures this voice in real time and sends the voice data to the server.

[0844] 2. The server converts the received voice data into text using Google Cloud Speech-to-Text. At this time, the voice command "Show news subtitles" is converted into text data "Show news subtitles."

[0845] 3. The server uses generative artificial intelligence (OpenAI GPT-4) to regenerate the text data into natural-sounding sentences. For example, if a user is watching the news on TV, the initial text "Show me the news subtitles" is generated into a detailed sentence: "News: The stock market rose significantly today." The input prompt for the generative AI model will be in the following text format:

[0846] Example prompt: "Regenerate this text into a meaningful sentence. Original text: 'In the news: The stock market rose significantly today.'"

[0847] 4. The server checks the generated text with a grammar checker (Grammarly API) and corrects any errors or unnatural points to generate the final text.

[0848] 5. The server converts the final text into speech using Amazon Polly, generating a natural-sounding speech, for example, "In the news: The stock market rose significantly today."

[0849] 6. The server sends the generated voice data and the final text to the terminal, which displays "In the news: The stock market rose significantly today" on the screen and plays the voice at the same time.

[0850] Specific examples

[0851] As a specific example, the system operation when the user utters "Display subtitles for the news" will be described.

[0852] 1. The user says, "Show me closed captions for the news."

[0853] 2. The device captures the audio and sends it to the server.

[0854] 3. The server converts the audio data into text.

[0855] 4. The server uses the generative AI model to regenerate the initial text into detailed sentences.

[0856] 5. The server checks the generated text with a grammar checker and corrects it.

[0857] 6. The server converts the final text into audio data.

[0858] 7. The server sends the voice data and text to the device.

[0859] 8. The device displays the final text and plays the audio.

[0860] In this way, the system of the present invention can convert speech into text and speech in real time, providing information to the user in a natural and easy-to-understand manner.

[0861] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0862] Step 1:

[0863] The process begins when a user speaks a command in front of the device. For example, they might say, "Display subtitles for the news." This voice is captured by the device's microphone and saved as audio data in real time. The input is audio data, and the output is the audio data saved on the device. The device then prepares to send this audio data to the server via the network.

[0864] Step 2:

[0865] The device transmits the captured audio data to the server via the network. The input here is the audio data stored on the device, and the output is the audio data to be transmitted to the server. The device establishes a network connection and executes communication processing to transfer the audio data to the server.

[0866] Step 3:

[0867] The server converts the voice data received from the device into text data using voice recognition technology such as Google Cloud Speech-to-Text. The input is the voice data received by the server, and the output is text data. The server analyzes the voice data and performs the specific operation of calling a voice recognition API to convert it into text.

[0868] Step 4:

[0869] The server uses generative artificial intelligence (e.g., OpenAI GPT-4) to regenerate the initial text into a meaningful sentence. The input is text data obtained by speech recognition technology, and the output is the regenerated sentence. The specific format of the prompt sentence is, "Please regenerate this text into a meaningful sentence. Original text: 'News: The stock market rose significantly today.'" The server sends the prompt sentence to the generative AI model and obtains the generated sentence.

[0870] Step 5:

[0871] The server checks the generated text with a grammar checker (e.g., Grammarly API) and corrects grammar and punctuation. The input is the text generated by the generative AI model, and the output is the final text after corrections. The server sends the text to the grammar checker, receives correction feedback, and incorporates it.

[0872] Step 6:

[0873] The server converts the final corrected text into audio data using text-to-speech technology such as Amazon Polly. The input is the final text and the output is audio data. The server calls the text-to-speech API to convert the text into audio and obtains the generated audio data.

[0874] Step 7:

[0875] The server sends the generated voice data and final text to the terminal. The input is the generated voice data and final text, and the output is the data to be sent to the terminal. The server performs communication processing to transfer the voice data and text to the terminal via the network.

[0876] Step 8:

[0877] The terminal displays the final text received from the server on the screen and simultaneously plays the audio data from the speaker. The input is the audio data and text data received from the server, and the output is the text display on the screen and audio playback from the speaker. Specifically, the terminal outputs the final text to the display and plays the audio data on the audio device.

[0878] By performing the above steps, the system of the present invention can convert speech into text and speech in real time, and provide information to the user in a natural and easy-to-understand format.

[0879] (Application example 1)

[0880] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0881] Conventional content viewing systems lack a mechanism for providing real-time subtitles or summaries of the content users are watching, which results in a lack of an improved viewing experience for users, making it difficult to understand specific scenes or easily check the introduction of the next episode through audio and text.

[0882] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0883] In this invention, the server includes means for converting input speech into text, means for regenerating the generated text into meaningful sentences, means for converting the regenerated text into speech and outputting it, means for displaying the generated text on a display device, means for providing the output speech using a speech output device, means for summarizing the next content based on the generated text, and communication means for transmitting the generated text and speech data to a user terminal, thereby enabling the user to check subtitles and content summaries in both speech and text in real time.

[0884] The "means for converting input voice into text" refers to a means for analyzing voice data input by a user and converting the content of the voice data into text data of the corresponding sentence.

[0885] The "means for regenerating generated text into meaningful sentences" refers to a means for processing the text generated by the initial speech recognition to reconstruct it into natural sentences based on context and grammar.

[0886] The "means for converting the regenerated text into speech and outputting the speech" refers to a means for converting the regenerated text data into speech data and performing processing to output the speech data to the user.

[0887] "Means for displaying the generated text on a display device" refers to means for transmitting the generated text data to a display device such as a display or screen in order to visually display it, and displaying it in a form that can be confirmed by the user.

[0888] The "means for providing audio output using an audio output device" refers to a means for allowing a user to listen to the generated audio data via an audio output device such as a speaker or headphones.

[0889] The "means for summarizing the next content based on the generated text" is a processing means for briefly summarizing the next episode or important content based on the initial input speech and the regenerated text.

[0890] "Communication means for transmitting the generated text and voice data to the user's terminal" refers to a means including communication technology for transmitting the generated text and voice data from the server to a terminal such as the user's smartphone or head-mounted display.

[0891] System Overview

[0892] This invention is a system that provides subtitles and summaries of content in real time in response to user voice commands while viewing content. This system consists of three main elements: a server, a terminal, and a user, and these elements work in conjunction with each other.

[0893] Hardware and Software Configuration

[0894] server:

[0895] Speech recognition engine used: Google Speech-to-Text API

[0896] Generative artificial intelligence model used: OpenAI GPT model

[0897] Text-to-speech engine used: Amazon Polly

[0898] Communication method: Internet communication

[0899] Device:

[0900] Audio input device: Microphone (built into smartphone or head-mounted display)

[0901] Audio output device: Speakers or headphones

[0902] Display devices: smartphone and head-mounted display screens

[0903] Processing flow

[0904] When a user uses a device to give voice commands, the device's microphone captures the voice and sends it to a server over the Internet. The server converts the voice into text using a speech recognition engine (Google Speech-to-Text API). It then uses a generative artificial intelligence model (OpenAI GPT model) to regenerate the text into natural-sounding sentences appropriate to the context. The generated text is then converted into audio data by a text-to-speech engine (Amazon Polly) and sent back to the device over the Internet. Finally, the device displays the text on the screen and plays the audio through speakers or headphones.

[0905] Specific example explanation

[0906] Example 1: Displaying subtitles

[0907] While watching a movie, a user says, "Show subtitles." The device's microphone captures the audio and sends it to the server. The server converts the audio to text, regenerates it, converts it back to audio, and sends it to the device. The device displays on the screen, "Movie content: The protagonist sets off on a new adventure." and simultaneously plays the audio.

[0908] Example 2: Next episode introduction

[0909] While watching a drama, a user says, "Please tell me about the next episode." The device's microphone captures the audio and sends it to the server. The server converts the audio into text and regenerates it based on the context. The generated text reads, "In the next episode, the protagonist sets out on a new adventure." This is converted into audio and sent to the device. The device displays "In the next episode, the protagonist sets out on a new adventure" on the screen and simultaneously plays it aloud.

[0910] Prompt Sentence Examples

[0911] User voice: "Please introduce the next episode."

[0912] Translated text: "Please introduce the next episode"

[0913] Example of a generative AI model input prompt: "Please explain the content of the next episode."

[0914] Generated output: "In the next episode, our protagonist sets off on a new adventure, facing various obstacles and meeting new allies."

[0915] In this way, the system converts speech to text and displays and speaks the reproduced text, providing a user with an enhanced viewing experience.

[0916] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0917] Step 1:

[0918] The user gives a voice command. For example, the user says, "Please introduce the next episode." This becomes the voice input data.

[0919] Step 2:

[0920] The device's microphone captures the voice input data, which is then sent by the device to the server.

[0921] Step 3:

[0922] The server receives the voice data and converts it into text data using a speech recognition engine (Google Speech-to-Text API). The input in this step is voice data, and the output is text data.

[0923] Step 4:

[0924] The server uses a generative artificial intelligence model (OpenAI GPT model) to regenerate the converted text data into meaningful sentences. In this step, it takes the text data as input and generates natural-looking sentences that are appropriate for the context. The output is the regenerated text.

[0925] Step 5:

[0926] The server converts the regenerated text data into audio data using a text-to-speech engine (Amazon Polly). The input is the regenerated text, and the output is audio data.

[0927] Step 6:

[0928] The server sends the generated voice data and text data to the terminal, which outputs the voice data and text data.

[0929] Step 7:

[0930] The text data received by the terminal is displayed on the screen. At the same time, the audio data is played back using the terminal's audio output device (speaker or headphones). This allows the user to see the text on the screen and hear the content aloud. The input is text data and audio data, and the output is screen display and audio playback.

[0931] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0932] A specific example of the configuration of a system for implementing the present invention is shown below.

[0933] System Overview

[0934] This system not only converts speech to text, regenerates the text, and converts the regenerated text back into speech for output, but also recognizes the user's emotions and reflects them in the output content, providing a more comprehensive and personalized viewing experience. The specific processing flow and functions of this system are described below.

[0935] System configuration

[0936] This system consists of three elements: a server, a terminal, and a user. The system operates according to the following procedure.

[0937] Audio collection and text conversion

[0938] 1. Collecting voice input

[0939] The user speaks the audio of the television broadcast into the microphone of the terminal, for example, "display the subtitles of the news" or the actual content of the television news broadcast.

[0940] The terminal captures this audio in real time and transmits the audio data as is to the server.

[0941] 2. Speech-to-text

[0942] The server converts the received voice data into text data using voice recognition technology. For example, it converts the voice "Show me the subtitles for the news" into the text "Show me the subtitles for the news."

[0943] Text semantic generation and refinement

[0944] 3. Text Analysis and Generation

[0945] The server uses generative artificial intelligence to regenerate meaningful sentences based on the initial text generated by speech recognition technology. At this time, the server takes into account context and grammar to generate natural-sounding sentences. For example, it generates a sentence like, "Today's news is that the stock market has risen significantly."

[0946] 4. Text Tweaks

[0947] The server checks the generated text for grammar, punctuation, etc. and makes any necessary adjustments. The final text is "Today's news is that the stock market has risen significantly."

[0948] User Emotion Recognition

[0949] 5. Emotion collection and analysis

[0950] The device uses an emotion recognition engine to analyze the user's facial expressions and tone of voice during voice input. This analysis is performed in real time and emotion data is generated.

[0951] The server receives the emotion data and reflects it in the generated text and voice.

[0952] Text-to-speech conversion and output

[0953] 6. Text to speech conversion

[0954] The server then converts the final text data into audio data using text-to-speech technology, with the tone and speed of the generated audio adjusted to match the user's emotions.

[0955] 7. Text and audio transmission and output

[0956] The server transmits the generated voice data and the final text to the terminal.

[0957] The terminal displays the final text "Today's news is that the stock market has risen significantly" in scrolling format on the screen.

[0958] The device then plays the audio data back to the user through an audio output device (speaker or headphones). The audio is tailored to the user's emotions, providing a more personalized listening experience.

[0959] Specific examples

[0960] For example, if a user stands in front of a TV and says, "Display subtitles for the news," with a surprised expression on their face, the system operates as follows.

[0961] 1. The user looks surprised while saying, "Show me the news subtitles."

[0962] 2. The device captures the voice and facial expressions and sends the voice and emotion data to the server.

[0963] 3. The server uses speech recognition technology to convert the text into "Show me the news subtitles."

[0964] 4. The server uses generative artificial intelligence to recreate the broadcast: "In the news: The stock market rose significantly today."

[0965] 5. The server adjusts the text and tone of the voice based on the emotional data.

[0966] 6. The server sends the final text and adjusted audio data to the device.

[0967] 7. The device displays the caption "In the news: The stock market rose significantly today" on the screen and plays a voiceover in a calibrated tone of surprise.

[0968] In this way, the system converts speech to text and then converts the regenerated text into speech that matches the user's emotions, providing a personalized listening experience.

[0969] The processing flow will be explained below.

[0970] Step 1:

[0971] The user speaks the audio of the television broadcast into the device's microphone, for example, saying "display the news subtitles" or the actual content of the television news broadcast.

[0972] Step 2:

[0973] The device captures the user's voice in real time through a microphone and acquires it as voice data, while also collecting emotional information such as the user's facial expressions and tone of voice.

[0974] Step 3:

[0975] The device transmits the acquired voice data and emotion data to a server via the Internet.

[0976] Step 4:

[0977] The server converts the received voice data into text data using voice recognition technology. For example, it converts the voice "Show me the subtitles for the news" into the text "Show me the subtitles for the news."

[0978] Step 5:

[0979] The server uses generative artificial intelligence to regenerate meaningful sentences based on the initial text generated by speech recognition technology. At this time, it takes into account context and grammar to generate natural sentences. For example, the generated sentence might be, "Today's news is that the stock market has risen significantly."

[0980] Step 6:

[0981] The server checks the generated text for grammar, punctuation, etc. and makes any necessary adjustments. The final text is "Today's news is that the stock market has risen significantly."

[0982] Step 7:

[0983] The server analyzes the received emotional data to recognize the user's emotional state, which includes facial expressions such as joy, surprise, and sadness, as well as tone of voice.

[0984] Step 8:

[0985] The server adjusts the text and audio output depending on the user's emotion. For example, if the emotion is "surprise," it will emphasize the text, raise the tone of the audio, and increase the playback speed.

[0986] Step 9:

[0987] The server converts the final text into audio data using text-to-speech technology, and the generated audio is adjusted in tone and speed to match the user's emotions.

[0988] Step 10:

[0989] The server sends the adjusted voice data and the final text to the terminal.

[0990] Step 11:

[0991] The terminal displays the final text "Today's news is that the stock market has risen significantly" in scrolling format on the screen.

[0992] Step 12:

[0993] The device then plays the audio data back to the user through an audio output device (speaker or headphones). The audio is tailored to the user's emotions, providing a more personalized listening experience.

[0994] Example 2

[0995] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0996] Conventional speech recognition systems often convert input speech into text and output the text as is. This makes it difficult to provide a personalized listening experience that reflects the user's emotions. Furthermore, the generated text often lacks proper grammar and tone adjustment, which can detract from a natural conversational experience. Furthermore, the visual display and audio output are not integrated, preventing comprehensive information provision.

[0997] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0998] In this invention, the server includes means for converting input speech into text, means for regenerating the generated text into meaningful sentences, means for recognizing a user's emotion and adjusting the tone and speed of the regenerated text and speech based on the emotion, means for converting the regenerated text into speech and outputting it, means for displaying the final text in a scrolling format on a display device, and means for providing the output speech to the user using an audio output device, thereby enabling a natural conversation experience and a personalized listening and listening experience that is tailored to the user's emotion.

[0999] The "means for converting input speech into text" refers to a system or technology that analyzes speech data input by a user and converts it into corresponding text data.

[1000] A "means for regenerating generated text into meaningful sentences" is a technology or system that uses initial text data to reconstruct sentences that have natural meaning, taking into account context and grammar.

[1001] "Means for recognizing user emotions" refers to technology or a system that analyzes the user's facial expressions and tone of voice to understand their emotional state at that time.

[1002] The "means for converting the reproduced text into speech and outputting it" refers to a technology or system that analyzes text data, converts it into corresponding speech data, and plays it back to the user.

[1003] The "means for displaying in a scrolling format on a display device" refers to a technology or system for visually displaying text data in a format that moves continuously on the display device.

[1004] "Speech recognition technology" is a technology that analyzes a voice signal and generates corresponding text data.

[1005] A "generative AI model" is a model that uses artificial intelligence to generate meaningful sentences from given text data.

[1006] An "emotion recognition engine" is a technology or system that analyzes a user's facial expressions and tone of voice to identify their emotional state.

[1007] "Text-to-speech technology" is a technology that analyzes text data and generates speech data based on its content.

[1008] An "audio output device" is a device or system that reproduces audio data and allows the user to listen to it.

[1009] The system of the present invention converts voice input into text, then regenerates the text into meaningful sentences, and further recognizes the user's emotions to adjust the tone and speed of the generated text and speech, providing a more comprehensive and personalized listening experience. This system consists of three elements: a server, a terminal, and a user, and operates according to the following process flow.

[1010] Audio collection and text conversion

[1011] The user speaks the audio of the television broadcast into the microphone of the terminal, for example, saying "Display subtitles for the news."

[1012] The device captures this voice in real time and generates digital audio data, which is then immediately sent to the server.

[1013] The server converts the received voice data into text data using speech recognition technology (commonly known as a speech recognition engine). A major speech recognition technology is Google Cloud Speech-to-Text. The converted text data is used in the next step.

[1014] Text semantic generation and refinement

[1015] The server uses a generative AI model (commonly known as a language generation model) to regenerate meaningful sentences based on the initial text generated by speech recognition. This series of processes uses GPT-3 and other technologies. Specifically, based on the text obtained through speech recognition, "Show me the subtitles for the news," it generates a natural-sounding sentence that is in line with the context, "Today's news is that the stock market has risen significantly."

[1016] After the text is generated, the server checks it for grammar and punctuation and makes any necessary adjustments, resulting in the final text sounding natural: "Today's news is that the stock market has risen significantly."

[1017] User Emotion Recognition

[1018] The device uses an emotion recognition engine (commonly known as an emotion analysis system) to analyze the user's facial expressions and tone of voice when they input voice. This analysis is performed in real time, and user emotion data is generated. A major emotion recognition engine is the Microsoft Azure Emotion API.

[1019] The server receives the emotion data and reflects it in the generated text and voice. Specifically, it adjusts the tone of the text or speeds up the voice depending on the emotion of surprise.

[1020] Text-to-speech conversion and output

[1021] The server then converts the final text data into audio data using text-to-speech technology (commonly known as a text-to-speech engine). Amazon Polly is used for this conversion. The generated audio is adjusted in tone and speed to match the user's emotion. For example, if surprise is recognized, a voice will be generated saying, "Today's news is that the stock market has risen significantly," in a surprised tone.

[1022] The server sends the generated voice data and the final text to the terminal. The terminal displays the final text, "Today's news is that the stock market has risen significantly," in a scrolling format on the screen. The terminal also plays the sent voice data through an audio output device (e.g., speaker or headphones). This allows the news to be conveyed in a tone that corresponds to the emotion.

[1023] Specific example explanation

[1024] The operation when a user is in front of a television and says "Display subtitles for the news" while making a surprised expression is as follows.

[1025] 1. The user looks surprised while saying, "Show me the news subtitles."

[1026] 2. The device captures the voice and facial expressions and sends the voice and emotion data to the server.

[1027] 3. The server uses speech recognition technology to convert the audio "Show me the news subtitles" into the text "Show me the news subtitles."

[1028] 4. The server uses the generative AI model to regenerate the broadcast content: "Today's news is that the stock market has risen significantly."

[1029] 5. The server adjusts the text and tone of the voice based on the emotional data.

[1030] 6. The server sends the final text and adjusted audio data to the device.

[1031] 7. The device displays the caption "Today's news: The stock market has risen significantly" on the screen and plays a voiceover in a calibrated surprised tone.

[1032] Prompt Sentence Examples

[1033] Below are some example prompts to input to a generative artificial intelligence model:

[1034] "Convert the user's speech, 'Show me the news subtitles,' into text and generate a natural-sounding sentence based on the speech. Also, convert the generated sentence into speech and adjust the tone and speed according to the user's level of surprise."

[1035] By combining these processes, the present invention makes it possible to realize a system that analyzes the voice input by the user, reproduces it as natural sentences, and provides a viewing experience that is in line with the user's emotions.

[1036] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1037] Step 1:

[1038] The user speaks into the microphone of the device to reproduce the audio of a television broadcast. For example, they say, "Show me the subtitles for the news." The input at this time is voice data.

[1039] Specifically, the user speaks into a microphone, which captures the speech.

[1040] Step 2:

[1041] The device captures the spoken voice in real time and transmits it to the server as digital voice data. At this time, the input is the voice acquired from the user in step 1, and the output is the digital voice data.

[1042] Specifically, the microphone captures the audio, converts the analog audio signal into a digital signal, and transmits it to a server via a network.

[1043] Step 3:

[1044] The server converts the received voice data into text data using speech recognition technology. The input here is digital voice data, and the output is text data. The technology used is a speech recognition engine (e.g., Google Cloud Speech-to-Text).

[1045] Specifically, the voice data is analyzed and converted into a corresponding character string.

[1046] Step 4:

[1047] The server uses a generative AI model (e.g., GPT-3) to regenerate the initial text data generated by speech recognition into natural, meaningful sentences. At this time, the input is the initial text data, and the output is the generated natural sentences.

[1048] Specifically, the initial text is input into a generative AI model, which then generates meaningful sentences taking into account context and grammar.

[1049] Step 5:

[1050] The server checks the grammar and punctuation of the generated text and fine-tunes it if necessary. The input is the regenerated text data, and the output is the final fine-tuned text data.

[1051] Specifically, the grammar check tool is used to detect and correct punctuation and grammatical errors.

[1052] Step 6:

[1053] The device analyzes the user's facial expression and tone of voice during voice input using an emotion recognition engine (e.g., Microsoft Azure Emotion API). The input is the user's facial expression and tone of voice, and the output is emotional data.

[1054] Specifically, it uses a camera and microphone to capture the user's facial expressions and tone of voice, and then analyzes the data in real time to recognize emotions.

[1055] Step 7:

[1056] The server adjusts the tone of the text and voice based on the emotion data. The input is the emotion data and the fine-tuned text data, and the output is the adjusted voice data and text data.

[1057] Specifically, it analyzes emotional data and appropriately adjusts the tone and speed of the generated text and audio based on the results.

[1058] Step 8:

[1059] The server converts the adjusted text data into audio data using text-to-speech technology (e.g., Amazon Polly), where the input is the adjusted text data and the output is audio data.

[1060] Specifically, the process involves using text-to-speech technology to convert text data into natural spoken voice.

[1061] Step 9:

[1062] The server transmits the generated voice data and the final text data to the terminal. The input here is the voice data and the text data, and the output is the data transmission to the terminal.

[1063] Specifically, the voice data and text data are transferred to the terminal via the network using a data transmission protocol.

[1064] Step 10:

[1065] The terminal displays the final text in scrolling format on the screen and plays the audio to the user through an audio output device (e.g., speaker or headphones). The input is the final text data and audio data, and the output is the provision of visual and auditory information to the user.

[1066] Specifically, the operation includes a screen control process for displaying text in a scrolling format and an audio playback process for controlling an audio playback device.

[1067] In this way, the system analyzes the user's speech, adjusts the generated text and audio to the user's emotions, and finally provides visual and audio information.

[1068] (Application example 2)

[1069] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1070] Conventional speech recognition systems can convert input speech into text and reproduce it, but they cannot provide a personalized viewing experience that reflects the user's emotions. Furthermore, in order to provide a comfortable interaction in real time, it is necessary to appropriately reflect emotional fluctuations. The present invention aims to solve these problems and provide a more comprehensive and personalized user experience.

[1071] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for converting input speech into text, means for regenerating the generated text into meaningful sentences, means for converting the regenerated text into speech and outputting it, and means for recognizing the user's emotion at the time of speech input and adjusting the tone of the generated text and speech based on that emotion. This enables dynamic text generation and speech output based on the user's emotion.

[1072] The "means for converting input speech into text" refers to a means for converting speech data into text data using speech recognition technology.

[1073] "Means for regenerating generated text into meaningful sentences" refers to means for using generative artificial intelligence to regenerate initial text data into natural-looking sentences that take into account context and grammar.

[1074] The "means for converting the regenerated text into speech and outputting it" refers to a means for converting the final text data into speech data using text-to-speech conversion technology and outputting it as speech to the user.

[1075] "Means for recognizing the user's emotions during voice input and adjusting the tone of the generated text and voice based on those emotions" refers to means for analyzing the user's facial expressions and tone of voice in real time and adjusting the tone of the text and voice based on the analysis results in accordance with the user's emotions.

[1076] This system converts speech input into text, regenerates it in a meaningful form, and finally outputs the regenerated text in a voice adjusted based on the user's emotions. This system consists of three elements: a server, a terminal, and a user.

[1077] System configuration

[1078] Collecting voice input and converting it to text

[1079] The user speaks into the microphone in the vehicle. The device captures this voice in real time and sends the voice data to the server. The server then converts the voice data into text data using voice recognition technology (e.g., Google Cloud Speech-to-Text).

[1080] Text analysis and generation

[1081] The server uses a generative artificial intelligence model (e.g., OpenAI's GPT-4) to regenerate the initial text data into meaningful sentences that take into account context and grammar, using prompts based on specific requests entered by the user.

[1082] An example prompt is:

[1083] Please make the user speak in natural sentences: Start navigation

[1084] Emotion collection and analysis

[1085] The device uses the vehicle's smart cameras and microphones to analyze the user's facial expressions and tone of voice in real time, and the analyzed emotional data is sent to a server, which then uses this data to adjust the tone and speed of the generated text and voice according to the user's emotions.

[1086] Text-to-speech conversion and output

[1087] The server converts the final text data into audio data using text-to-speech technology (e.g., Google Cloud Text-to-Speech). The tone and speed of the audio are adjusted based on the user's emotions. The adjusted audio and final text are then sent to the device, which outputs the audio through its speakers and displays the final text on the screen. This allows the user to enjoy a personalized listening experience that matches their emotions.

[1088] Specific examples

[1089] For example, consider a situation where a driver says, "Start navigation." If the system detects that the driver is tired, it generates the following sentence: "Start navigation to your destination. This road has little traffic." It then adjusts the tone of the generated sentence to be gentler. This adjusted voice is played through the vehicle's speakers, and the screen displays, "Start navigation to your destination. This road has little traffic."

[1090] This provides responses that are in line with the user's emotional state, making interactions more natural and engaging.

[1091] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1092] Step 1:

[1093] The device collects the user's voice input using a microphone. The input voice data is sent directly to the server. This allows the voice data to reach the server in real time.

[1094] Step 2:

[1095] The server converts the received voice data into text data using speech recognition technology (e.g., Google Cloud Speech-to-Text). The input here is voice data, and the output is the corresponding text data. Through this conversion process, the user's speech is represented in text format.

[1096] Step 3:

[1097] The server uses a generative artificial intelligence model (e.g., OpenAI's GPT-4) to regenerate the initial text data into meaningful sentences. The input is the initial text obtained by speech recognition technology, and the output is a regenerated natural sentence. Here, a prompt sentence is used to generate an appropriate response.

[1098] Step 4:

[1099] The terminal analyzes the user's facial expressions and tone of voice in real time using the smart camera and microphone in the vehicle. This collects the user's emotional data and sends it to the server. The input of this step is the user's facial and vocal data from the camera and microphone, and the output is emotional data.

[1100] Step 5:

[1101] The server adjusts the tone and speed of the generated text and voice based on the obtained emotion data according to the user's emotion. The input is emotion data and regenerated text data, and the output is text and voice data adjusted according to the emotion. This adjustment enables natural responses that are in line with the user's emotion.

[1102] Step 6:

[1103] The server converts the final text data into audio data using text-to-speech technology (e.g., Google Cloud Text-to-Speech), where the input is the adjusted text data and the output is the adjusted audio data.

[1104] Step 7:

[1105] The terminal plays the adjusted audio data sent from the server using an audio output device (speaker), and displays the final text data on the screen. The input is the adjusted audio data and text data, and the output is audio output and text display. This allows the user to have a personalized viewing experience that matches their emotions.

[1106] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1107] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1108] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1109] [Fourth embodiment]

[1110] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1111] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1112] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1113] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1114] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1115] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1116] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1117] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1118] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1119] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1120] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1121] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1122] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1123] A specific example of the configuration of a system for implementing the present invention is shown below.

[1124] System Overview

[1125] This system converts speech to text, regenerates the text, and converts the regenerated text back into speech for output, providing a comprehensive viewing experience. The specific processing flow and functions of this system are described below.

[1126] System configuration

[1127] This system consists of three elements: a server, a terminal, and a user. The system operates according to the following procedure.

[1128] Audio collection and text conversion

[1129] 1. Collecting voice input

[1130] The user inputs the audio of the television broadcast through the microphone of the device, for example, by issuing a voice command such as "display subtitles for the news" or by speaking the actual content of the television news broadcast.

[1131] The terminal captures this audio in real time and transmits the audio data as is to the server.

[1132] 2. Speech-to-text

[1133] The server converts the received voice data into text data using speech recognition technology. For example, a voice saying "Show me the subtitles for the news" is converted into text "Show me the subtitles for the news."

[1134] Text semantic generation and refinement

[1135] 3. Text Analysis and Generation

[1136] The server uses generative artificial intelligence to regenerate meaningful sentences based on the initial text generated by speech recognition technology, taking into account context and grammar to generate natural-sounding sentences. For example, the generated sentence might read, "In the news: The stock market rose significantly today."

[1137] 4. Text Tweaks

[1138] The server checks the generated text for grammar and punctuation and makes corrections if necessary. The final text is "In the news: The stock market rose significantly today."

[1139] Text-to-speech conversion and output

[1140] 5. Text to speech conversion

[1141] The server converts the final text into audio data using text-to-speech technology, and the resulting audio is played back as a natural voice.

[1142] 6. Text and audio transmission and output

[1143] The server transmits the generated voice data and the final text to the terminal.

[1144] The device displays subtitles on the screen, such as "In the news: The stock market saw a big jump today," while simultaneously playing audio data, allowing the user to consume both audio and text information.

[1145] Specific examples

[1146] For example, if a user says "Display subtitles for the news" in front of a TV, the system operates as follows.

[1147] 1. The user says, "Show me closed captions for the news."

[1148] 2. The device captures the audio and sends it to the server.

[1149] 3. The server uses speech recognition technology to convert the text into "Show me the news subtitles."

[1150] 4. The server uses generative artificial intelligence to recreate the broadcast: "In the news: The stock market rose significantly today."

[1151] 5. The server then converts this text into audio data using text-to-speech technology.

[1152] 6. The server sends the generated voice data and text to the terminal.

[1153] 7. The device will display the caption "In the news: The stock market saw a big jump today" on the screen and simultaneously play the audio.

[1154] In this way, the system converts speech to text, regenerates the text into meaningful sentences, and converts the regenerated text into speech for output, providing a comprehensive listening experience.

[1155] The processing flow will be explained below.

[1156] Step 1:

[1157] The user speaks the audio of the television broadcast into the device's microphone, for example, saying "display the news subtitles" or the actual content of the television news broadcast.

[1158] Step 2:

[1159] The terminal captures the user's voice in real time through a microphone and acquires it as voice data.

[1160] Step 3:

[1161] The device transmits the acquired voice data to a server via the Internet.

[1162] Step 4:

[1163] The server converts the received voice data into text data using voice recognition technology. For example, it converts the voice "Show me the subtitles for the news" into the text "Show me the subtitles for the news."

[1164] Step 5:

[1165] The server uses generative artificial intelligence to regenerate the initial text obtained from the speech recognition technology into a meaningful sentence based on the context, for example, "Today's news is that the stock market has risen significantly."

[1166] Step 6:

[1167] The server checks the generated text for grammar, punctuation, etc. and makes any necessary adjustments. The final text is "Today's news is that the stock market has risen significantly."

[1168] Step 7:

[1169] The server converts the final text data into audio data using text-to-speech technology, and the generated audio can be played back as a natural voice.

[1170] Step 8:

[1171] The server transmits the generated voice data and the final text to the terminal.

[1172] Step 9:

[1173] The terminal displays the final text "Today's news is that the stock market has risen significantly" in scrolling format on the screen.

[1174] Step 10:

[1175] The terminal reproduces the transmitted audio data to the user through an audio output device (speaker or headphones), allowing the user to use both audio information and text information.

[1176] Example 1

[1177] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1178] Conventional speech recognition systems simply convert speech to text without correcting the meaning or grammar of the text. This can result in unnatural and difficult-to-understand text for users. Furthermore, systems that convert the converted text back into speech and output it are limited. The present invention aims to solve these problems by providing a comprehensive system that converts speech to text, regenerates the text into natural-sounding sentences, and then re-outputs it as speech.

[1179] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1180] In this invention, the server includes means for converting input speech into text, means for regenerating the generated text into meaningful sentences, means for converting the regenerated text into speech and outputting it, means for capturing input speech in real time, means for checking the generated text with a grammar checker and correcting it if necessary, and means for providing the generated speech data and final text by screen display and audio playback, thereby enabling users to use natural and easy-to-understand text and speech information in real time.

[1181] "Speech recognition technology" is a technology that analyzes input voice data and converts it into text data.

[1182] "Generative AI" is an AI technology that generates natural-looking sentences based on input initial text, taking into account context and grammar.

[1183] "Text-to-speech technology" is a technology that converts text data into voice data and outputs it as natural voice.

[1184] "Capturing in real time" means that input audio data can be processed immediately as it is.

[1185] A "grammar checker" is a tool or technique for checking the grammar and punctuation of generated text and correcting errors.

[1186] "Displaying on screen" means displaying the generated text on a display device such as a monitor or screen.

[1187] "Audio playback" refers to playing back the generated audio data through an audio output device such as a speaker.

[1188] "Means" refers to a method, technique, or device used to achieve a particular purpose.

[1189] A "server" is a computer system connected to a network that processes requests from multiple terminals and provides data.

[1190] "Terminal" means a device that allows a user to input voice and display / play back the generated text and voice.

[1191] "User" refers to a person using the system to provide voice input.

[1192] A "network" is a communications infrastructure that allows multiple computer systems to communicate with each other.

[1193] An "input prompt" refers to the text of instructions or questions given to a generative artificial intelligence.

[1194] System Overview

[1195] The system of the present invention converts speech into text, regenerates the text into meaningful sentences, and finally converts the regenerated text into speech and outputs it. The system consists of three elements: a server, a terminal, and a user, and is realized using the following hardware and software.

[1196] Hardware and software used

[1197] 1. Hardware

[1198] Terminal: A device equipped with a microphone, display, and speaker that allows a user to input voice and output generated text and voice.

[1199] Server: Equipped with the computing resources to perform speech recognition, generative artificial intelligence, and text-to-speech conversion.

[1200] 2. Software

[1201] Speech recognition technology: Using Google Cloud Speech-to-Text, etc., voice data is converted into text data.

[1202] Generative artificial intelligence: Using models such as OpenAI GPT-4, initial text is generated into meaningful sentences.

[1203] Grammar checker: Checks and corrects the grammar of the generated text using APIs such as Grammarly.

[1204] Text-to-speech technology: Using technologies such as Amazon Polly, text data is converted into natural-sounding speech data.

[1205] System Operation

[1206] 1. The user stands in front of the device and gives a voice command, such as "Show subtitles for the news." The device's microphone captures this voice in real time and sends the voice data to the server.

[1207] 2. The server converts the received voice data into text using Google Cloud Speech-to-Text. At this time, the voice command "Show news subtitles" is converted into text data "Show news subtitles."

[1208] 3. The server uses generative artificial intelligence (OpenAI GPT-4) to regenerate the text data into natural-sounding sentences. For example, if a user is watching the news on TV, the initial text "Show me the news subtitles" is generated into a detailed sentence: "News: The stock market rose significantly today." The input prompt for the generative AI model will be in the following text format:

[1209] Example prompt: "Regenerate this text into a meaningful sentence. Original text: 'In the news: The stock market rose significantly today.'"

[1210] 4. The server checks the generated text with a grammar checker (Grammarly API) and corrects any errors or unnatural points to generate the final text.

[1211] 5. The server converts the final text into speech using Amazon Polly, generating a natural-sounding speech, for example, "In the news: The stock market rose significantly today."

[1212] 6. The server sends the generated voice data and the final text to the terminal, which displays "In the news: The stock market rose significantly today" on the screen and plays the voice at the same time.

[1213] Specific examples

[1214] As a specific example, the system operation when the user utters "Display subtitles for the news" will be described.

[1215] 1. The user says, "Show me closed captions for the news."

[1216] 2. The device captures the audio and sends it to the server.

[1217] 3. The server converts the audio data into text.

[1218] 4. The server uses the generative AI model to regenerate the initial text into detailed sentences.

[1219] 5. The server checks the generated text with a grammar checker and corrects it.

[1220] 6. The server converts the final text into audio data.

[1221] 7. The server sends the voice data and text to the device.

[1222] 8. The device displays the final text and plays the audio.

[1223] In this way, the system of the present invention can convert speech into text and speech in real time, providing information to the user in a natural and easy-to-understand manner.

[1224] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1225] Step 1:

[1226] The process begins when a user speaks a command in front of the device. For example, they might say, "Display subtitles for the news." This voice is captured by the device's microphone and saved as audio data in real time. The input is audio data, and the output is the audio data saved on the device. The device then prepares to send this audio data to the server via the network.

[1227] Step 2:

[1228] The device transmits the captured audio data to the server via the network. The input here is the audio data stored on the device, and the output is the audio data to be transmitted to the server. The device establishes a network connection and executes communication processing to transfer the audio data to the server.

[1229] Step 3:

[1230] The server converts the voice data received from the device into text data using voice recognition technology such as Google Cloud Speech-to-Text. The input is the voice data received by the server, and the output is text data. The server analyzes the voice data and performs the specific operation of calling a voice recognition API to convert it into text.

[1231] Step 4:

[1232] The server uses generative artificial intelligence (e.g., OpenAI GPT-4) to regenerate the initial text into a meaningful sentence. The input is text data obtained by speech recognition technology, and the output is the regenerated sentence. The specific format of the prompt sentence is, "Please regenerate this text into a meaningful sentence. Original text: 'News: The stock market rose significantly today.'" The server sends the prompt sentence to the generative AI model and obtains the generated sentence.

[1233] Step 5:

[1234] The server checks the generated text with a grammar checker (e.g., Grammarly API) and corrects grammar and punctuation. The input is the text generated by the generative AI model, and the output is the final text after corrections. The server sends the text to the grammar checker, receives correction feedback, and incorporates it.

[1235] Step 6:

[1236] The server converts the final corrected text into audio data using text-to-speech technology such as Amazon Polly. The input is the final text and the output is audio data. The server calls the text-to-speech API to convert the text into audio and obtains the generated audio data.

[1237] Step 7:

[1238] The server sends the generated voice data and final text to the terminal. The input is the generated voice data and final text, and the output is the data to be sent to the terminal. The server performs communication processing to transfer the voice data and text to the terminal via the network.

[1239] Step 8:

[1240] The terminal displays the final text received from the server on the screen and simultaneously plays the audio data from the speaker. The input is the audio data and text data received from the server, and the output is the text display on the screen and audio playback from the speaker. Specifically, the terminal outputs the final text to the display and plays the audio data on the audio device.

[1241] By performing the above steps, the system of the present invention can convert speech into text and speech in real time, and provide information to the user in a natural and easy-to-understand format.

[1242] (Application example 1)

[1243] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1244] Conventional content viewing systems lack a mechanism for providing real-time subtitles or summaries of the content users are watching, which results in a lack of an improved viewing experience for users, making it difficult to understand specific scenes or easily check the introduction of the next episode through audio and text.

[1245] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1246] In this invention, the server includes means for converting input speech into text, means for regenerating the generated text into meaningful sentences, means for converting the regenerated text into speech and outputting it, means for displaying the generated text on a display device, means for providing the output speech using a speech output device, means for summarizing the next content based on the generated text, and communication means for transmitting the generated text and speech data to a user terminal, thereby enabling the user to check subtitles and content summaries in both speech and text in real time.

[1247] The "means for converting input voice into text" refers to a means for analyzing voice data input by a user and converting the content of the voice data into text data of the corresponding sentence.

[1248] The "means for regenerating generated text into meaningful sentences" refers to a means for processing the text generated by the initial speech recognition to reconstruct it into natural sentences based on context and grammar.

[1249] The "means for converting the regenerated text into speech and outputting the speech" refers to a means for converting the regenerated text data into speech data and performing processing to output the speech data to the user.

[1250] "Means for displaying the generated text on a display device" refers to means for transmitting the generated text data to a display device such as a display or screen in order to visually display it, and displaying it in a form that can be confirmed by the user.

[1251] The "means for providing audio output using an audio output device" refers to a means for allowing a user to listen to the generated audio data via an audio output device such as a speaker or headphones.

[1252] The "means for summarizing the next content based on the generated text" is a processing means for briefly summarizing the next episode or important content based on the initial input speech and the regenerated text.

[1253] "Communication means for transmitting the generated text and voice data to the user's terminal" refers to a means including communication technology for transmitting the generated text and voice data from the server to a terminal such as the user's smartphone or head-mounted display.

[1254] System Overview

[1255] This invention is a system that provides subtitles and summaries of content in real time in response to user voice commands while viewing content. This system consists of three main elements: a server, a terminal, and a user, and these elements work in conjunction with each other.

[1256] Hardware and Software Configuration

[1257] server:

[1258] Speech recognition engine used: Google Speech-to-Text API

[1259] Generative artificial intelligence model used: OpenAI GPT model

[1260] Text-to-speech engine used: Amazon Polly

[1261] Communication method: Internet communication

[1262] Device:

[1263] Audio input device: Microphone (built into smartphone or head-mounted display)

[1264] Audio output device: Speakers or headphones

[1265] Display devices: smartphone and head-mounted display screens

[1266] Processing flow

[1267] When a user uses a device to give voice commands, the device's microphone captures the voice and sends it to a server over the Internet. The server converts the voice into text using a speech recognition engine (Google Speech-to-Text API). It then uses a generative artificial intelligence model (OpenAI GPT model) to regenerate the text into natural-sounding sentences appropriate to the context. The generated text is then converted into audio data by a text-to-speech engine (Amazon Polly) and sent back to the device over the Internet. Finally, the device displays the text on the screen and plays the audio through speakers or headphones.

[1268] Specific example explanation

[1269] Example 1: Displaying subtitles

[1270] While watching a movie, a user says, "Show subtitles." The device's microphone captures the audio and sends it to the server. The server converts the audio to text, regenerates it, converts it back to audio, and sends it to the device. The device displays on the screen, "Movie content: The protagonist sets off on a new adventure." and simultaneously plays the audio.

[1271] Example 2: Next episode introduction

[1272] While watching a drama, a user says, "Please tell me about the next episode." The device's microphone captures the audio and sends it to the server. The server converts the audio into text and regenerates it based on the context. The generated text reads, "In the next episode, the protagonist sets out on a new adventure." This is converted into audio and sent to the device. The device displays "In the next episode, the protagonist sets out on a new adventure" on the screen and simultaneously plays it aloud.

[1273] Prompt Sentence Examples

[1274] User voice: "Please introduce the next episode."

[1275] Translated text: "Please introduce the next episode"

[1276] Example of a generative AI model input prompt: "Please explain the content of the next episode."

[1277] Generated output: "In the next episode, our protagonist sets off on a new adventure, facing various obstacles and meeting new allies."

[1278] In this way, the system converts speech to text and displays and speaks the reproduced text, providing a user with an enhanced viewing experience.

[1279] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1280] Step 1:

[1281] The user gives a voice command. For example, the user says, "Please introduce the next episode." This becomes the voice input data.

[1282] Step 2:

[1283] The device's microphone captures the voice input data, which is then sent by the device to the server.

[1284] Step 3:

[1285] The server receives the voice data and converts it into text data using a speech recognition engine (Google Speech-to-Text API). The input in this step is voice data, and the output is text data.

[1286] Step 4:

[1287] The server uses a generative artificial intelligence model (OpenAI GPT model) to regenerate the converted text data into meaningful sentences. In this step, it takes the text data as input and generates natural-looking sentences that are appropriate for the context. The output is the regenerated text.

[1288] Step 5:

[1289] The server converts the regenerated text data into audio data using a text-to-speech engine (Amazon Polly). The input is the regenerated text, and the output is audio data.

[1290] Step 6:

[1291] The server sends the generated voice data and text data to the terminal, which outputs the voice data and text data.

[1292] Step 7:

[1293] The text data received by the terminal is displayed on the screen. At the same time, the audio data is played back using the terminal's audio output device (speaker or headphones). This allows the user to see the text on the screen and hear the content aloud. The input is text data and audio data, and the output is screen display and audio playback.

[1294] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1295] A specific example of the configuration of a system for implementing the present invention is shown below.

[1296] System Overview

[1297] This system not only converts speech to text, regenerates the text, and converts the regenerated text back into speech for output, but also recognizes the user's emotions and reflects them in the output content, providing a more comprehensive and personalized viewing experience. The specific processing flow and functions of this system are described below.

[1298] System configuration

[1299] This system consists of three elements: a server, a terminal, and a user. The system operates according to the following procedure.

[1300] Audio collection and text conversion

[1301] 1. Collecting voice input

[1302] The user speaks the audio of the television broadcast into the microphone of the terminal, for example, "display the subtitles of the news" or the actual content of the television news broadcast.

[1303] The terminal captures this audio in real time and transmits the audio data as is to the server.

[1304] 2. Speech-to-text

[1305] The server converts the received voice data into text data using voice recognition technology. For example, it converts the voice "Show me the subtitles for the news" into the text "Show me the subtitles for the news."

[1306] Text semantic generation and refinement

[1307] 3. Text Analysis and Generation

[1308] The server uses generative artificial intelligence to regenerate meaningful sentences based on the initial text generated by speech recognition technology. At this time, the server takes into account context and grammar to generate natural-sounding sentences. For example, it generates a sentence like, "Today's news is that the stock market has risen significantly."

[1309] 4. Text Tweaks

[1310] The server checks the generated text for grammar, punctuation, etc. and makes any necessary adjustments. The final text is "Today's news is that the stock market has risen significantly."

[1311] User Emotion Recognition

[1312] 5. Emotion collection and analysis

[1313] The device uses an emotion recognition engine to analyze the user's facial expressions and tone of voice during voice input. This analysis is performed in real time and emotion data is generated.

[1314] The server receives the emotion data and reflects it in the generated text and voice.

[1315] Text-to-speech conversion and output

[1316] 6. Text to speech conversion

[1317] The server then converts the final text data into audio data using text-to-speech technology, with the tone and speed of the generated audio adjusted to match the user's emotions.

[1318] 7. Text and audio transmission and output

[1319] The server transmits the generated voice data and the final text to the terminal.

[1320] The terminal displays the final text "Today's news is that the stock market has risen significantly" in scrolling format on the screen.

[1321] The device then plays the audio data back to the user through an audio output device (speaker or headphones). The audio is tailored to the user's emotions, providing a more personalized listening experience.

[1322] Specific examples

[1323] For example, if a user stands in front of a TV and says, "Display subtitles for the news," with a surprised expression on their face, the system operates as follows.

[1324] 1. The user looks surprised while saying, "Show me the news subtitles."

[1325] 2. The device captures the voice and facial expressions and sends the voice and emotion data to the server.

[1326] 3. The server uses speech recognition technology to convert the text into "Show me the news subtitles."

[1327] 4. The server uses generative artificial intelligence to recreate the broadcast: "In the news: The stock market rose significantly today."

[1328] 5. The server adjusts the text and tone of the voice based on the emotional data.

[1329] 6. The server sends the final text and adjusted audio data to the device.

[1330] 7. The device displays the caption "In the news: The stock market rose significantly today" on the screen and plays a voiceover in a calibrated tone of surprise.

[1331] In this way, the system converts speech to text and then converts the regenerated text into speech that matches the user's emotions, providing a personalized listening experience.

[1332] The processing flow will be explained below.

[1333] Step 1:

[1334] The user speaks the audio of the television broadcast into the device's microphone, for example, saying "display the news subtitles" or the actual content of the television news broadcast.

[1335] Step 2:

[1336] The device captures the user's voice in real time through a microphone and acquires it as voice data, while also collecting emotional information such as the user's facial expressions and tone of voice.

[1337] Step 3:

[1338] The device transmits the acquired voice data and emotion data to a server via the Internet.

[1339] Step 4:

[1340] The server converts the received voice data into text data using voice recognition technology. For example, it converts the voice "Show me the subtitles for the news" into the text "Show me the subtitles for the news."

[1341] Step 5:

[1342] The server uses generative artificial intelligence to regenerate meaningful sentences based on the initial text generated by speech recognition technology. At this time, it takes into account context and grammar to generate natural sentences. For example, the generated sentence might be, "Today's news is that the stock market has risen significantly."

[1343] Step 6:

[1344] The server checks the generated text for grammar, punctuation, etc. and makes any necessary adjustments. The final text is "Today's news is that the stock market has risen significantly."

[1345] Step 7:

[1346] The server analyzes the received emotional data to recognize the user's emotional state, which includes facial expressions such as joy, surprise, and sadness, as well as tone of voice.

[1347] Step 8:

[1348] The server adjusts the text and audio output depending on the user's emotion. For example, if the emotion is "surprise," it will emphasize the text, raise the tone of the audio, and increase the playback speed.

[1349] Step 9:

[1350] The server converts the final text into audio data using text-to-speech technology, and the generated audio is adjusted in tone and speed to match the user's emotions.

[1351] Step 10:

[1352] The server sends the adjusted voice data and the final text to the terminal.

[1353] Step 11:

[1354] The terminal displays the final text "Today's news is that the stock market has risen significantly" in scrolling format on the screen.

[1355] Step 12:

[1356] The device then plays the audio data back to the user through an audio output device (speaker or headphones). The audio is tailored to the user's emotions, providing a more personalized listening experience.

[1357] Example 2

[1358] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1359] Conventional speech recognition systems often convert input speech into text and output the text as is. This makes it difficult to provide a personalized listening experience that reflects the user's emotions. Furthermore, the generated text often lacks proper grammar and tone adjustment, which can detract from a natural conversational experience. Furthermore, the visual display and audio output are not integrated, preventing comprehensive information provision.

[1360] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1361] In this invention, the server includes means for converting input speech into text, means for regenerating the generated text into meaningful sentences, means for recognizing a user's emotion and adjusting the tone and speed of the regenerated text and speech based on the emotion, means for converting the regenerated text into speech and outputting it, means for displaying the final text in a scrolling format on a display device, and means for providing the output speech to the user using an audio output device, thereby enabling a natural conversation experience and a personalized listening and listening experience that is tailored to the user's emotion.

[1362] The "means for converting input speech into text" refers to a system or technology that analyzes speech data input by a user and converts it into corresponding text data.

[1363] A "means for regenerating generated text into meaningful sentences" is a technology or system that uses initial text data to reconstruct sentences that have natural meaning, taking into account context and grammar.

[1364] "Means for recognizing user emotions" refers to technology or a system that analyzes the user's facial expressions and tone of voice to understand their emotional state at that time.

[1365] The "means for converting the reproduced text into speech and outputting it" refers to a technology or system that analyzes text data, converts it into corresponding speech data, and plays it back to the user.

[1366] The "means for displaying in a scrolling format on a display device" refers to a technology or system for visually displaying text data in a format that moves continuously on the display device.

[1367] "Speech recognition technology" is a technology that analyzes a voice signal and generates corresponding text data.

[1368] A "generative AI model" is a model that uses artificial intelligence to generate meaningful sentences from given text data.

[1369] An "emotion recognition engine" is a technology or system that analyzes a user's facial expressions and tone of voice to identify their emotional state.

[1370] "Text-to-speech technology" is a technology that analyzes text data and generates speech data based on its content.

[1371] An "audio output device" is a device or system that reproduces audio data and allows the user to listen to it.

[1372] The system of the present invention converts voice input into text, then regenerates the text into meaningful sentences, and further recognizes the user's emotions to adjust the tone and speed of the generated text and speech, providing a more comprehensive and personalized listening experience. This system consists of three elements: a server, a terminal, and a user, and operates according to the following process flow.

[1373] Audio collection and text conversion

[1374] The user speaks the audio of the television broadcast into the microphone of the terminal, for example, saying "Display subtitles for the news."

[1375] The device captures this voice in real time and generates digital audio data, which is then immediately sent to the server.

[1376] The server converts the received voice data into text data using speech recognition technology (commonly known as a speech recognition engine). A major speech recognition technology is Google Cloud Speech-to-Text. The converted text data is used in the next step.

[1377] Text semantic generation and refinement

[1378] The server uses a generative AI model (commonly known as a language generation model) to regenerate meaningful sentences based on the initial text generated by speech recognition. This series of processes uses GPT-3 and other technologies. Specifically, based on the text obtained through speech recognition, "Show me the subtitles for the news," it generates a natural-sounding sentence that is in line with the context, "Today's news is that the stock market has risen significantly."

[1379] After the text is generated, the server checks it for grammar and punctuation and makes any necessary adjustments, resulting in the final text sounding natural: "Today's news is that the stock market has risen significantly."

[1380] User Emotion Recognition

[1381] The device uses an emotion recognition engine (commonly known as an emotion analysis system) to analyze the user's facial expressions and tone of voice when they input voice. This analysis is performed in real time, and user emotion data is generated. A major emotion recognition engine is the Microsoft Azure Emotion API.

[1382] The server receives the emotion data and reflects it in the generated text and voice. Specifically, it adjusts the tone of the text or speeds up the voice depending on the emotion of surprise.

[1383] Text-to-speech conversion and output

[1384] The server then converts the final text data into audio data using text-to-speech technology (commonly known as a text-to-speech engine). Amazon Polly is used for this conversion. The generated audio is adjusted in tone and speed to match the user's emotion. For example, if surprise is recognized, a voice will be generated saying, "Today's news is that the stock market has risen significantly," in a surprised tone.

[1385] The server sends the generated voice data and the final text to the terminal. The terminal displays the final text, "Today's news is that the stock market has risen significantly," in a scrolling format on the screen. The terminal also plays the sent voice data through an audio output device (e.g., speaker or headphones). This allows the news to be conveyed in a tone that corresponds to the emotion.

[1386] Specific example explanation

[1387] The operation when a user is in front of a television and says "Display subtitles for the news" while making a surprised expression is as follows.

[1388] 1. The user looks surprised while saying, "Show me the news subtitles."

[1389] 2. The device captures the voice and facial expressions and sends the voice and emotion data to the server.

[1390] 3. The server uses speech recognition technology to convert the audio "Show me the news subtitles" into the text "Show me the news subtitles."

[1391] 4. The server uses the generative AI model to regenerate the broadcast content: "Today's news is that the stock market has risen significantly."

[1392] 5. The server adjusts the text and tone of the voice based on the emotional data.

[1393] 6. The server sends the final text and adjusted audio data to the device.

[1394] 7. The device displays the caption "Today's news: The stock market has risen significantly" on the screen and plays a voiceover in a calibrated surprised tone.

[1395] Prompt Sentence Examples

[1396] Below are some example prompts to input to a generative artificial intelligence model:

[1397] "Convert the user's speech, 'Show me the news subtitles,' into text and generate a natural-sounding sentence based on the speech. Also, convert the generated sentence into speech and adjust the tone and speed according to the user's level of surprise."

[1398] By combining these processes, the present invention makes it possible to realize a system that analyzes the voice input by the user, reproduces it as natural sentences, and provides a viewing experience that is in line with the user's emotions.

[1399] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1400] Step 1:

[1401] The user speaks into the microphone of the device to reproduce the audio of a television broadcast. For example, they say, "Show me the subtitles for the news." The input at this time is voice data.

[1402] Specifically, the user speaks into a microphone, which captures the speech.

[1403] Step 2:

[1404] The device captures the spoken voice in real time and transmits it to the server as digital voice data. At this time, the input is the voice acquired from the user in step 1, and the output is the digital voice data.

[1405] Specifically, the microphone captures the audio, converts the analog audio signal into a digital signal, and transmits it to a server via a network.

[1406] Step 3:

[1407] The server converts the received voice data into text data using speech recognition technology. The input here is digital voice data, and the output is text data. The technology used is a speech recognition engine (e.g., Google Cloud Speech-to-Text).

[1408] Specifically, the voice data is analyzed and converted into a corresponding character string.

[1409] Step 4:

[1410] The server uses a generative AI model (e.g., GPT-3) to regenerate the initial text data generated by speech recognition into natural, meaningful sentences. At this time, the input is the initial text data, and the output is the generated natural sentences.

[1411] Specifically, the initial text is input into a generative AI model, which then generates meaningful sentences taking into account context and grammar.

[1412] Step 5:

[1413] The server checks the grammar and punctuation of the generated text and fine-tunes it if necessary. The input is the regenerated text data, and the output is the final fine-tuned text data.

[1414] Specifically, the grammar check tool is used to detect and correct punctuation and grammatical errors.

[1415] Step 6:

[1416] The device analyzes the user's facial expression and tone of voice during voice input using an emotion recognition engine (e.g., Microsoft Azure Emotion API). The input is the user's facial expression and tone of voice, and the output is emotional data.

[1417] Specifically, it uses a camera and microphone to capture the user's facial expressions and tone of voice, and then analyzes the data in real time to recognize emotions.

[1418] Step 7:

[1419] The server adjusts the tone of the text and voice based on the emotion data. The input is the emotion data and the fine-tuned text data, and the output is the adjusted voice data and text data.

[1420] Specifically, it analyzes emotional data and appropriately adjusts the tone and speed of the generated text and audio based on the results.

[1421] Step 8:

[1422] The server converts the adjusted text data into audio data using text-to-speech technology (e.g., Amazon Polly), where the input is the adjusted text data and the output is audio data.

[1423] Specifically, the process involves using text-to-speech technology to convert text data into natural spoken voice.

[1424] Step 9:

[1425] The server transmits the generated voice data and the final text data to the terminal. The input here is the voice data and the text data, and the output is the data transmission to the terminal.

[1426] Specifically, the voice data and text data are transferred to the terminal via the network using a data transmission protocol.

[1427] Step 10:

[1428] The terminal displays the final text in scrolling format on the screen and plays the audio to the user through an audio output device (e.g., speaker or headphones). The input is the final text data and audio data, and the output is the provision of visual and auditory information to the user.

[1429] Specifically, the operation includes a screen control process for displaying text in a scrolling format and an audio playback process for controlling an audio playback device.

[1430] In this way, the system analyzes the user's speech, adjusts the generated text and audio to the user's emotions, and finally provides visual and audio information.

[1431] (Application example 2)

[1432] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1433] Conventional speech recognition systems can convert input speech into text and reproduce it, but they cannot provide a personalized viewing experience that reflects the user's emotions. Furthermore, in order to provide a comfortable interaction in real time, it is necessary to appropriately reflect emotional fluctuations. The present invention aims to solve these problems and provide a more comprehensive and personalized user experience.

[1434] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for converting input speech into text, means for regenerating the generated text into meaningful sentences, means for converting the regenerated text into speech and outputting it, and means for recognizing the user's emotion at the time of speech input and adjusting the tone of the generated text and speech based on that emotion. This enables dynamic text generation and speech output based on the user's emotion.

[1435] The "means for converting input speech into text" refers to a means for converting speech data into text data using speech recognition technology.

[1436] "Means for regenerating generated text into meaningful sentences" refers to means for using generative artificial intelligence to regenerate initial text data into natural-looking sentences that take into account context and grammar.

[1437] The "means for converting the regenerated text into speech and outputting it" refers to a means for converting the final text data into speech data using text-to-speech conversion technology and outputting it as speech to the user.

[1438] "Means for recognizing the user's emotions during voice input and adjusting the tone of the generated text and voice based on those emotions" refers to means for analyzing the user's facial expressions and tone of voice in real time and adjusting the tone of the text and voice based on the analysis results in accordance with the user's emotions.

[1439] This system converts speech input into text, regenerates it in a meaningful form, and finally outputs the regenerated text in a voice adjusted based on the user's emotions. This system consists of three elements: a server, a terminal, and a user.

[1440] System configuration

[1441] Collecting voice input and converting it to text

[1442] The user speaks into the microphone in the vehicle. The device captures this voice in real time and sends the voice data to the server. The server then converts the voice data into text data using voice recognition technology (e.g., Google Cloud Speech-to-Text).

[1443] Text analysis and generation

[1444] The server uses a generative artificial intelligence model (e.g., OpenAI's GPT-4) to regenerate the initial text data into meaningful sentences that take into account context and grammar, using prompts based on specific requests entered by the user.

[1445] An example prompt is:

[1446] Please make the user speak in natural sentences: Start navigation

[1447] Emotion collection and analysis

[1448] The device uses the vehicle's smart cameras and microphones to analyze the user's facial expressions and tone of voice in real time, and the analyzed emotional data is sent to a server, which then uses this data to adjust the tone and speed of the generated text and voice according to the user's emotions.

[1449] Text-to-speech conversion and output

[1450] The server converts the final text data into audio data using text-to-speech technology (e.g., Google Cloud Text-to-Speech). The tone and speed of the audio are adjusted based on the user's emotions. The adjusted audio and final text are then sent to the device, which outputs the audio through its speakers and displays the final text on the screen. This allows the user to enjoy a personalized listening experience that matches their emotions.

[1451] Specific examples

[1452] For example, consider a situation where a driver says, "Start navigation." If the system detects that the driver is tired, it generates the following sentence: "Start navigation to your destination. This road has little traffic." It then adjusts the tone of the generated sentence to be gentler. This adjusted voice is played through the vehicle's speakers, and the screen displays, "Start navigation to your destination. This road has little traffic."

[1453] This provides responses that are in line with the user's emotional state, making interactions more natural and engaging.

[1454] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1455] Step 1:

[1456] The device collects the user's voice input using a microphone. The input voice data is sent directly to the server. This allows the voice data to reach the server in real time.

[1457] Step 2:

[1458] The server converts the received voice data into text data using speech recognition technology (e.g., Google Cloud Speech-to-Text). The input here is voice data, and the output is the corresponding text data. Through this conversion process, the user's speech is represented in text format.

[1459] Step 3:

[1460] The server uses a generative artificial intelligence model (e.g., OpenAI's GPT-4) to regenerate the initial text data into meaningful sentences. The input is the initial text obtained by speech recognition technology, and the output is a regenerated natural sentence. Here, a prompt sentence is used to generate an appropriate response.

[1461] Step 4:

[1462] The terminal analyzes the user's facial expressions and tone of voice in real time using the smart camera and microphone in the vehicle. This collects the user's emotional data and sends it to the server. The input of this step is the user's facial and vocal data from the camera and microphone, and the output is emotional data.

[1463] Step 5:

[1464] The server adjusts the tone and speed of the generated text and voice based on the obtained emotion data according to the user's emotion. The input is emotion data and regenerated text data, and the output is text and voice data adjusted according to the emotion. This adjustment enables natural responses that are in line with the user's emotion.

[1465] Step 6:

[1466] The server converts the final text data into audio data using text-to-speech technology (e.g., Google Cloud Text-to-Speech), where the input is the adjusted text data and the output is the adjusted audio data.

[1467] Step 7:

[1468] The terminal plays the adjusted audio data sent from the server using an audio output device (speaker), and displays the final text data on the screen. The input is the adjusted audio data and text data, and the output is audio output and text display. This allows the user to have a personalized viewing experience that matches their emotions.

[1469] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1470] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1471] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1472] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1473] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1474] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1475] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1476] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1477] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1478] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1479] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1480] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1481] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1482] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1483] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1484] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1485] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1486] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1487] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1488] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1489] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1490] The following is further disclosed regarding the above embodiment.

[1491] (Claim 1)

[1492] a means for converting input speech into text;

[1493] a means for regenerating the generated text into meaningful sentences;

[1494] means for converting the regenerated text into speech and outputting the speech;

[1495] A system including:

[1496] (Claim 2)

[1497] a means for utilizing speech recognition technology to convert input speech into text;

[1498] a means for utilizing generative artificial intelligence to regenerate the generated text into meaningful sentences;

[1499] a means for utilizing text-to-speech technology to convert the regenerated text into speech;

[1500] 10. The system of claim 1, comprising:

[1501] (Claim 3)

[1502] A means for capturing input speech in real time and converting it to text;

[1503] means for displaying the generated text in a scrolling format on a display device;

[1504] means for providing a user with audio output using an audio output device;

[1505] 10. The system of claim 1, comprising:

[1506] "Example 1"

[1507] (Claim 1)

[1508] a means for converting input speech into text;

[1509] a means for regenerating the generated text into meaningful sentences;

[1510] means for converting the regenerated text into speech and outputting the speech;

[1511] a means for capturing input audio in real time;

[1512] A means to run the generated text through a grammar checker and correct it if necessary;

[1513] means for providing the generated voice data and final text by screen display and voice playback;

[1514] A system including:

[1515] (Claim 2)

[1516] a means for utilizing speech recognition technology to convert input speech into text;

[1517] a means for utilizing generative artificial intelligence to regenerate the generated text into meaningful sentences;

[1518] a means for utilizing text-to-speech technology to convert the regenerated text into speech;

[1519] A means to run the generated text through a grammar checker and correct it if necessary;

[1520] 10. The system of claim 1, comprising:

[1521] (Claim 3)

[1522] A means for capturing input speech in real time and converting it to text;

[1523] means for displaying the generated text on a display device and reproducing the audio using an audio output device;

[1524] means for transmitting the input voice to a server via a network;

[1525] means for providing the generated voice data and final text by screen display and voice playback;

[1526] 10. The system of claim 1, comprising:

[1527] "Application Example 1"

[1528] (Claim 1)

[1529] a means for converting input speech into text;

[1530] a means for regenerating the generated text into meaningful sentences;

[1531] means for converting the regenerated text into speech and outputting the speech;

[1532] means for displaying the generated text on a display device;

[1533] means for providing an output sound using an audio output device;

[1534] A system including:

[1535] (Claim 2)

[1536] a means for utilizing speech recognition technology to convert input speech into text;

[1537] a means for utilizing generative artificial intelligence to regenerate the generated text into meaningful sentences;

[1538] a means for utilizing text-to-speech technology to convert the regenerated text into speech;

[1539] A means for summarizing the following based on the generated text:

[1540] communication means for transmitting the generated text and voice data to a user terminal;

[1541] 10. The system of claim 1, comprising:

[1542] (Claim 3)

[1543] A means for capturing input speech in real time and converting it to text;

[1544] means for displaying the generated text in a scrolling format on a display device;

[1545] means for providing a user with audio output using an audio output device;

[1546] A means for summarizing content based on the generated text and providing it in a form that is easy for the user to understand;

[1547] 10. The system of claim 1, comprising:

[1548] "Example 2: Combining Emotion Engines"

[1549] (Claim 1)

[1550] a means for converting input speech into text;

[1551] a means for regenerating the generated text into meaningful sentences;

[1552] means for recognizing a user's emotion and adjusting the tone and speed of the reproduced text or speech based on the emotion;

[1553] means for converting the regenerated text into speech and outputting the speech;

[1554] means for displaying the final text in a scrolling format on a display device;

[1555] means for providing a user with audio output using an audio output device;

[1556] A system including:

[1557] (Claim 2)

[1558] a means for utilizing speech recognition technology to convert input speech into text;

[1559] A means for utilizing a generative AI model to regenerate the generated text into meaningful sentences; and

[1560] a means for utilizing an emotion recognition engine to analyze the emotion of a user;

[1561] a means for utilizing text-to-speech technology to convert the regenerated text into speech;

[1562] means for adjusting the tone and rate of the generated speech to match the user's emotions;

[1563] 10. The system of claim 1, comprising:

[1564] (Claim 3)

[1565] a means for capturing input voice and a user's facial expression in real time to obtain voice data and emotion data;

[1566] means for displaying the generated text in a scrolling format on a display device;

[1567] means for providing the user with the audio output using the adjusted audio output device;

[1568] 10. The system of claim 1, comprising:

[1569] "Application example 2 when combining emotion engines"

[1570] (Claim 1)

[1571] a means for converting input speech into text;

[1572] a means for regenerating the generated text into meaningful sentences;

[1573] means for converting the regenerated text into speech and outputting the speech;

[1574] means for recognizing a user's emotion during speech input and adjusting the tone of the generated text and speech based on the emotion;

[1575] A system including:

[1576] (Claim 2)

[1577] a means for utilizing speech recognition technology to convert input speech into text;

[1578] a means for utilizing generative artificial intelligence to regenerate the generated text into meaningful sentences;

[1579] a means for utilizing text-to-speech technology to convert the regenerated text into speech;

[1580] a means for analyzing user sentiment and adjusting the tone and rate of text and speech based on that data;

[1581] 10. The system of claim 1, comprising:

[1582] (Claim 3)

[1583] A means for capturing input speech in real time and converting it to text;

[1584] means for displaying the generated text in a scrolling format on a display device;

[1585] means for providing a user with audio output using an audio output device;

[1586] means for collecting emotion data of a user and transmitting the collected emotion data to a server;

[1587] means for outputting a voice in a tone corresponding to the emotion of the user using the emotion data;

[1588] 10. The system of claim 1, comprising: [Explanation of symbols]

[1589] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. a means for converting input speech into text; a means for regenerating the generated text into meaningful sentences; means for converting the regenerated text into speech and outputting the speech; A system including:

2. a means for utilizing speech recognition technology to convert input speech into text; a means for utilizing generative artificial intelligence to regenerate the generated text into meaningful sentences; a means for utilizing text-to-speech technology to convert the regenerated text into speech; The system of claim 1 , comprising:

3. A means for capturing input speech in real time and converting it to text; means for displaying the generated text in a scrolling format on a display device; means for providing a user with audio output using an audio output device; The system of claim 1 , comprising:

Images