system

Abstract

A system is provided.SOLUTION: A system comprising: means for capturing audio of a television broadcast; means for sending captured audio data to an audio analysis API; means for displaying text data obtained from the audio analysis API on a user interface in real-time; means for receiving a user text selection and sending the selected text data to an audio generation API; and means for playing back audio data obtained from the audio generation API.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In modern society, watching television is an important means of obtaining information, but for people with hearing impairments and foreigners who have difficulty understanding Japanese, it is difficult to fully enjoy the information. For this reason, there is a need for technology that allows these users to easily understand the content of television broadcasts in real time. [Means for solving the problem]

[0005] The present invention provides a system that combines a speech analysis API that captures audio from television broadcasts and automatically converts it into text, and a speech generation API that converts text data into natural-sounding speech. The system includes the following means:

[0006] A way to capture audio from a TV broadcast

[0007] A means of sending captured voice data to a voice analysis API and obtaining text data.

[0008] A means for displaying the retrieved text data in a user interface in real time.

[0009] A means of receiving a user's text selection and sending that selected text data to the speech generation API

[0010] A means of playing back audio data obtained from the audio generation API

[0011] This makes it possible to provide television broadcast content in real time as text and audio information to people with hearing impairments and foreigners who have difficulty understanding Japanese.

[0012] A "means for capturing audio from television broadcasts" is a device or software that has the function of capturing audio signals output from a television and processing them as digital data.

[0013] The "means for sending to the voice analysis API" refers to a device or software that has the function of transferring captured voice data to a voice analysis service via a network and requesting voice analysis.

[0014] A "voice analysis API" is an application programming interface for analyzing input voice data and converting the content into text data.

[0015] The "means for displaying text data on a user interface in real time" refers to a device or software that has the function of sequentially displaying acquired text data on a screen that can be viewed by the user.

[0016] "User interface" is a general term for software or hardware that provides the input and output means for a user to interact with a system.

[0017] A "means for receiving a text selection and transmitting the selected text data to a speech generation API" is a device or software that has the functionality to accept a user's selection of a specific portion of displayed text data and transmit the selected text data to a speech generation service.

[0018] A "speech generation API" is an application programming interface for analyzing input text data and generating natural-sounding speech based on that content.

[0019] The "means for reproducing audio data" refers to a device or software that has the function of reproducing acquired audio data through an audio output device such as a speaker or headphones. [Brief explanation of the drawings]

[0020] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0021] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0022] First, the terms used in the following description will be explained.

[0023] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0024] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0025] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0026] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0027] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0028] [First embodiment]

[0029] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0030] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0031] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0032] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0033] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0034] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0035] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0036] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0037] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0038] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0039] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0040] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0041] Summary of the Invention

[0042] This system converts television broadcast audio into text in real time and displays the text on a user interface. It also has the ability to convert user-selected text into speech and play it back. This system is implemented using a speech analysis API and a speech generation API.

[0043] System Configuration

[0044] This system consists of three main components: a server, a terminal, and a user. The server is responsible for converting speech to text and generating speech, while the terminal captures speech, displays text, and plays back speech. The user operates the system through the terminal.

[0045] Overview of program processing

[0046] 1. Capture and transmit audio

[0047] Terminal

[0048] While the user is watching a television program, the terminal captures the television sound in real time using a built-in microphone or a connected external device.

[0049] The captured audio data is periodically sent to a server.

[0050] 2. Speech-to-text

[0051] server

[0052] The server receives the voice data sent from the device and sends it to the voice analysis API.

[0053] The voice analysis API analyzes voice data and converts it into text data.

[0054] The server receives the converted text data and formats it.

[0055] 3. Real-time subtitle generation and display

[0056] Terminal

[0057] The formatted text data is sent to the terminal and displayed in real time on the user interface.

[0058] 4. User's voice conversion request

[0059] User

[0060] The user selects a particular portion of the displayed text and requests to hear that text aloud.

[0061] Terminal

[0062] This request is sent to the server along with the text data.

[0063] 5. Text-to-speech

[0064] server

[0065] The server sends the text data selected by the user to the speech generation API.

[0066] The speech generation API analyzes text data and converts it into speech data.

[0067] The server receives the generated audio data.

[0068] 6. Playing audio data

[0069] Terminal

[0070] The audio data is sent to the terminal and played back via an audio player.

[0071] Specific examples

[0072] Example 1: For a hearing impaired user

[0073] Voice input

[0074] A user is watching a news program. During this time, the device captures the audio of the news and periodically transmits it to a server.

[0075] Text

[0076] The server sends the audio data of the news program to a speech analysis API and obtains text data such as, "Tonight's news will report on important events from around the world."

[0077] Real-time subtitle display

[0078] The captured text data is sent to the terminal and displayed on the user interface, allowing the user to read the subtitles displayed on the screen.

[0079] Example 2: For foreign users who do not understand Japanese

[0080] Voice input

[0081] A user is watching a variety show. The device captures the audio of the variety show and sends it to the server.

[0082] Text

[0083] The server sends the audio data from the variety show to the audio analysis API and obtains text data such as "This week's guest is a world-famous artist."

[0084] Voice conversion request

[0085] The user selects the subtitle portion, "This week's guest is a world-class artist," to hear it in English.

[0086] Voice generation

[0087] The server sends the selected text data to the speech generation API and obtains English speech data.

[0088] Audio playback

[0089] The terminal plays back the acquired English audio data, allowing the user to understand the content.

[0090] The above is an example of a specific embodiment of the system according to the present invention, which aims to meet the diverse needs of users and make television viewing more accessible.

[0091] The processing flow will be explained below.

[0092] Step 1:

[0093] The device captures the audio of the TV program being watched by the user in real time using a built-in microphone or an external device, and stores the captured audio data in a buffer.

[0094] Step 2:

[0095] The terminal transmits the buffered audio data to the server at regular time intervals (for example, every second) using a protocol such as an HTTP POST request.

[0096] Step 3:

[0097] The server prepares to process the audio data received from the device and creates a request to send the audio data to the audio analysis API.

[0098] Step 4:

[0099] The server sends the voice data to the voice analysis API, which converts the voice data into text data.

[0100] Step 5:

[0101] The server receives the text data returned from the speech analysis API, formats the received text data, and adds timestamps, etc., as needed.

[0102] Step 6:

[0103] The server sends the formatted text data to the device using a WebSocket connection or an HTTP response.

[0104] Step 7:

[0105] The terminal acquires the text data received from the server and displays the acquired text data as real-time subtitles on the user interface.

[0106] Step 8:

[0107] The user selects a specific portion of the displayed subtitles and clicks on the corresponding interface button when he or she wishes to hear the text spoken.

[0108] Step 9:

[0109] The terminal receives the user's selection operation and transmits the selected text data and a voice conversion request to the server.

[0110] Step 10:

[0111] The server creates a request to send the selected text data received from the device to the speech generation API.

[0112] Step 11:

[0113] The server sends the text data to the speech generation API and requests speech conversion, which converts the text data into speech data.

[0114] Step 12:

[0115] The server receives the voice data returned from the voice generation API and sends it to the device as is.

[0116] Step 13:

[0117] The terminal acquires the audio data received from the server and plays the acquired audio data using an audio player so that the user can hear it.

[0118] The above are the specific steps of the system's processing, and the specific operations at each step are described, allowing the system's operations to be clearly understood.

[0119] Example 1

[0120] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0121] There is a need for a system that can convert television audio into text in real time and provide it to users in an easy-to-understand manner. In particular, users with hearing impairments and foreign users who speak different languages ​​need assistance in understanding television content. There is also a need for a system that allows users to select specific text and listen to it aloud.

[0122] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0123] In this invention, the server includes means for capturing audio data of a television broadcast, means for transmitting the captured audio data to the server, means for the server to transmit the audio data to an audio analysis API and format text data obtained from the audio analysis API, means for displaying the formatted text data on a user interface in real time, means for receiving a text selection from a user and transmitting the selected text data to an audio generation API, and means for playing the audio data obtained from the audio generation API, thereby enabling real-time conversion of audio to text and further playback of the user-selected text by audio.

[0124] "Means for capturing audio from a television broadcast" refers to means, including built-in microphones and external devices, used to collect audio from a television broadcast in real time.

[0125] The "means for transmitting captured audio data to a server" refers to a means having a function for periodically dividing collected audio data into batches and transmitting the batches to a server.

[0126] The "means for the server to transmit voice data to the voice analysis API" refers to a means including a process of transmitting voice data received by the server to the voice analysis API and converting the voice data into text data.

[0127] "Means for formatting text data obtained from a speech analysis API" refers to means for formatting text data obtained from a speech analysis API and for separating sentences and breaking lines to make the data easier to read.

[0128] "Means for displaying formatted text data in a user interface in real time" refers to means having the function of displaying formatted text data in a user interface in real time, including web pages and mobile applications.

[0129] The "means for receiving a user's text selection and transmitting the selected text data to the speech generation API" refers to a means that, when a user selects a specific portion of displayed text, transmits the selected text data to the speech generation API.

[0130] "Means for playing back audio data obtained from an audio generation API" refers to means for playing back audio data obtained from an audio generation API on a terminal, and includes an audio player.

[0131] MODE FOR CARRYING OUT THE INVENTION

[0132] overview

[0133] This system converts television broadcast audio into text in real time and displays the text on a user interface. It also has the ability to convert user-selected text into speech and play it back. This system is implemented using a speech analysis API and a speech generation API.

[0134] Major hardware and software used

[0135] Hardware:

[0136] Device: Built-in microphone or external device (e.g. USB microphone)

[0137] software:

[0138] Speech analysis API: For example, Google Cloud Speech-to-Text API

[0139] Speech generation API: For example, Amazon Polly

[0140] User interface: HTML, JavaScript, and a front-end framework such as React or Vue.js

[0141] Example of operation

[0142] Audio capture and transmission

[0143] Terminal

[0144] While a user is watching a TV program, the device captures the TV audio in real time through a built-in microphone or an external device. Specifically, an audio capture application on the device monitors the microphone input and captures audio data in real time.

[0145] The captured audio data is periodically split into small batches and sent to the server, for example every 5 seconds.

[0146] Speech to text

[0147] server

[0148] The server receives the voice data sent from the device via an HTTP request, and sends the received voice data to a voice analysis API (e.g., Google Cloud Speech-to-Text API).

[0149] The speech analysis API converts the speech data into text data, and the server receives the converted text data and performs any necessary formatting, such as adding sentence breaks and line breaks to make the text easier to read.

[0150] Real-time subtitle display

[0151] Terminal

[0152] The formatted text data is sent from the server to the terminal. Specifically, the server sends the text data as an HTTP response.

[0153] The device displays this text data in a user interface (e.g., web page, mobile app) in real time. Using JavaScript or a front-end framework, the text can be displayed dynamically.

[0154] User's voice conversion request

[0155] User

[0156] The user selects a particular portion of the displayed text, for example by clicking on a block of text.

[0157] Terminal

[0158] Based on this selection, the text data and a request for speech conversion are sent to the server. Specifically, the selected text and the request type (e.g., speech conversion request) are sent to the server as a POST request.

[0159] Text-to-speech

[0160] server

[0161] The server receives the text data selected by the user and extracts it from the JSON body of the HTTP request.

[0162] Send the received text data to a speech generation API (e.g., Amazon Polly), authenticating with an API key and specifying the required parameters (e.g., language, tone of voice).

[0163] The speech generation API analyzes the text data and converts it into speech data, and the server receives the generated speech data.

[0164] Playing audio data

[0165] Terminal

[0166] The generated voice data is sent from the server to the terminal, usually as an HTTP response.

[0167] The device receives this audio data and plays it using an audio player (e.g. HTML5 Audio). JavaScript is used to set the URL of the audio data in the audio player, and the audio is played when the play button is pressed.

[0168] Examples of prompt statements

[0169] "Please explain in natural language a system that converts audio from a news program into text and plays back the text selected by the user as speech."

[0170] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0171] Step 1: Capture and send audio

[0172] Terminal

[0173] While the user is watching a TV program, the device captures TV audio in real time using a built-in microphone or an external device. The captured audio data is divided into batches every 5 seconds. The device periodically sends these audio data batches to the server. The input is the captured audio data, and the output is the batched audio data. This capture process is performed using the Audio Capture API, and transmission to the server is via an HTTP POST request.

[0174] Step 2: Speech to text

[0175] server

[0176] The server receives the audio data batches sent from the device. It sends the received audio data batches to a speech analysis API and converts the audio data into text data. Here, the Google Cloud Speech-to-Text API is used. The converted text data is received and formatted. Specifically, it adds sentence separators and line breaks to make it easier to read. The input is the audio data batches, and the output is formatted text data.

[0177] Step 3: Display real-time subtitles

[0178] Terminal

[0179] The terminal receives formatted text data sent from the server. The terminal displays this text data in the user interface in real time. The text is displayed dynamically using JavaScript and a front-end framework (e.g., React, Vue.js). The input is formatted text data, and the output is real-time subtitles displayed in the user interface.

[0180] Step 4: User's voice conversion request

[0181] User

[0182] The user selects a specific portion of the displayed text, for example by clicking on a block of text. The input is the text data displayed in real time, and the output is the text data selected by the user.

[0183] Step 5: Sending the selected text

[0184] Terminal

[0185] It receives the text data selected by the user and sends it to the server. Specifically, it sends the selected text data and the speech conversion request information to the server as a POST request. The input is the text data selected by the user, and the output is an HTTP request to the server.

[0186] Step 6: Convert text to speech

[0187] server

[0188] The server sends the received selected text data to the speech generation API. Amazon Polly is used here. Authentication is performed using an API key, and a speech conversion request is sent specifying the required parameters (e.g., language, tone of voice). The server receives the speech data obtained from the speech generation API. The input is the selected text data, and the output is the generated speech data.

[0189] Step 7: Playing back audio data

[0190] Terminal

[0191] Receives audio data sent from the server. The device receives the audio data and plays it using an audio player (e.g. HTML5 Audio). JavaScript is used to set the URL of the audio data in the audio player, and the audio is played when the play button is pressed. The input is the received audio data, and the output is the audio played to the user.

[0192] (Application example 1)

[0193] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0194] Conventional food delivery applications lack effective means of conveying audio information, such as restaurant promotional videos and menu introductions, to users. This poses a challenge, making it difficult to provide sufficient information to users with hearing impairments or who have difficulty understanding foreign languages. Furthermore, the lack of information provision through audiovisual interaction raises concerns about a decline in the quality of the user experience.

[0195] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0196] In this invention, the server includes means for transmitting voice data to a voice analysis API and acquiring text data, means for formatting the acquired text data and assigning a timestamp, means for displaying the text data on a user interface, means for receiving a user's text selection and transmitting it to a voice generation API to acquire voice data, means for playing the voice data, and means for capturing and converting the voice of an advertising medium into text to provide food delivery information, and converting the text selected by the user into voice and playing it back. This allows the user to visually check the voice information converted into text in real time, and further select the text as needed and play it back as voice information.

[0197] "Information media" refers to means of providing information through audio and video. This includes television broadcasts, radio broadcasts, online videos, etc.

[0198] A "voice analysis API" is an application programming interface (API) for analyzing voice data and converting it into text data. It is generally implemented using a cloud-based service.

[0199] A "user interface" is an interface that allows a user to interact with a system. It includes means such as a screen display and audio output.

[0200] A "Speech Generation API" is an application programming interface (API) for converting text data into speech data. It provides text-to-speech functionality.

[0201] "Audio data" means a digital representation of sound waves, typically captured by a microphone or recorded audio.

[0202] "Text data" refers to data written in characters. This includes text generated from audio data using a voice analysis API.

[0203] "Food delivery information" refers to information about food and drink delivery, including restaurant promotional videos, menu introductions, delivery status, etc.

[0204] "Promotional media" refers to the means of widely disseminating product information. This includes videos, audio, posters, etc.

[0205] This invention is a system that converts audio from television broadcasts and other information media into text in real time and displays it on a user interface. It also has the function of converting user-selected text into audio and playing it back. The following are specific examples of how this system can be implemented.

[0206] System Configuration

[0207] The system consists of three main components: a server, a terminal, and a user.

[0208] server

[0209] The server operates a program that uses the voice analysis API and voice generation API, which provides the following functions:

[0210] Send the audio data to the audio analysis API and obtain the text data.

[0211] The acquired text data is formatted and a timestamp is added.

[0212] The formatted text data is sent to the terminal and displayed.

[0213] It receives a text selection request from the user, sends the text to the speech generation API, and obtains the speech data.

[0214] Terminal

[0215] A terminal is a device (e.g., a smartphone or tablet) that is operated by a user. A terminal performs the following functions:

[0216] Capture audio using the built-in microphone or an external device.

[0217] Send the captured audio data to the server.

[0218] The text data transmitted from the server is displayed on the user interface in real time.

[0219] The text data selected by the user is sent again to the server, and the voice data is acquired.

[0220] The acquired audio data is played back.

[0221] User

[0222] The User is the final user of the system and performs the following operations:

[0223] Watch television broadcasts and other information media and capture audio through your device.

[0224] View real-time text data displayed in the user interface.

[0225] Select a specific portion of text and make a speech conversion request.

[0226] Listen to the audio being played and understand the information.

[0227] Program processing procedure

[0228] The server first receives the voice data sent from the device. Next, it sends the voice data to a voice analysis API to obtain text data. This text data is formatted and a timestamp is added. The formatted text is sent to the device and displayed in the user interface. When the user selects a portion of the text and requests voice conversion, the server sends the text data to a voice generation API to obtain the voice data. Finally, the voice data is sent to the device and played.

[0229] Hardware and software used

[0230] Hardware

[0231] Smartphones, tablets, and microphones

[0232] software

[0233] Speech analysis API (e.g., Google Cloud Speech-to-Text)

[0234] Speech generation APIs (e.g., Google Text-to-Speech)

[0235] Server-side program (implemented in Python etc.)

[0236] User Interface (using a front-end framework)

[0237] Specific examples

[0238] When a user uses a food delivery app, the audio from a restaurant's promotional video is captured by the smartphone's microphone. This audio is converted into text in real time and displayed on the user interface. The user selects the part of the displayed text that says "Today's special is sushi pizza" and requests that it be played aloud. The server sends this text to a speech generation API, which then transmits the generated audio to the device. Finally, the user hears the audio saying "sushi pizza."

[0239] Prompt Sentence Examples

[0240] "When a user types "sushi pizza" into your application, convert the text into speech and play it back."

[0241] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0242] Output results according to processing steps

[0243] Step 1:

[0244] The device captures audio from information media using a built-in microphone or an external device, and this audio data is collected in real time and sent to a server for further processing.

[0245] Input: Audio data (audio from information media)

[0246] Output: Audio data is sent to the server

[0247] Step 2:

[0248] The server receives the voice data sent from the device, then uses a voice analysis API to convert the voice data into text data, which is then formatted and timestamped.

[0249] Input: Audio data

[0250] Output: Formatted text data (with timestamp)

[0251] Step 3:

[0252] The server sends the formatted text data to the terminal.

[0253] Input: Formatted text data

[0254] Output: Text data is sent to the terminal

[0255] Step 4:

[0256] The terminal displays the transmitted text data on the user interface in real time, allowing the user to check the text data displayed in a visible format.

[0257] Input: Text data

[0258] Output: Real-time displayed text data

[0259] Step 5:

[0260] The user selects the desired part from the text data, and the selected text is sent back to the server as a speech generation request.

[0261] Input: Real-time displayed text data

[0262] Output: The selected text data is sent to the server.

[0263] Step 6:

[0264] The server uses a speech generation API to convert the selected text data into speech data, which is then sent to the device.

[0265] Input: Selected text data

[0266] Output: Generated audio data

[0267] Step 7:

[0268] The terminal plays the audio data that has been sent, and the user can hear the audio corresponding to the selected text.

[0269] Input: Generated audio data

[0270] Output: Played audio data (user hears the audio)

[0271] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0272] Summary of the Invention

[0273] This invention is a system that converts television broadcast audio into text in real time and displays that text on a user interface. It also has the ability to convert user-selected text into speech and play it back. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, the user experience can be further improved. This system is realized using a speech analysis API and a speech generation API.

[0274] System Configuration

[0275] This system consists of three main components: a server, a terminal, and a user. The server is responsible for converting speech to text and generating speech, while the terminal is responsible for capturing speech, displaying text, playing speech, and analyzing emotions using an emotion engine. The user operates the system through the terminal.

[0276] Overview of program processing

[0277] 1. Capture and transmit audio

[0278] Terminal

[0279] While the user is watching a television program, the terminal captures the television sound in real time using a built-in microphone or a connected external device.

[0280] The captured audio data is periodically sent to a server.

[0281] 2. Speech-to-text

[0282] server

[0283] The server receives the voice data sent from the device and sends it to the voice analysis API.

[0284] The voice analysis API analyzes voice data and converts it into text data.

[0285] The server receives the converted text data and formats it.

[0286] 3. Real-time subtitle generation and display

[0287] Terminal

[0288] The formatted text data is sent to the terminal and displayed in real time on the user interface.

[0289] 4. User's voice conversion request

[0290] User

[0291] The user selects a particular portion of the displayed text and requests to hear that text aloud.

[0292] Terminal

[0293] This request is sent to the server along with the text data.

[0294] 5. Text-to-speech

[0295] server

[0296] The server sends the text data selected by the user to the speech generation API.

[0297] The speech generation API analyzes text data and converts it into speech data.

[0298] The server receives the generated audio data.

[0299] 6. Playing audio data

[0300] Terminal

[0301] The audio data is sent to the terminal and played back via an audio player.

[0302] 7. Emotion Analysis Using an Emotion Engine

[0303] Terminal

[0304] The device captures the user's facial expressions or tone of voice in real time and sends them to the emotion engine.

[0305] The emotion engine analyzes the user's emotions and returns the results to the terminal.

[0306] 8. Emotion-Based Interface Adjustment

[0307] Terminal

[0308] Based on the analysis results from the emotion engine, the content of the text data displayed and the tone and tempo of the audio data are adjusted.

[0309] Specific examples

[0310] Example 1: For a hearing impaired user

[0311] Voice input

[0312] A user is watching a news program. During this time, the device captures the audio of the news and periodically transmits it to a server.

[0313] Text

[0314] The server sends the audio data of the news program to a speech analysis API and obtains text data such as, "Tonight's news will report on important events from around the world."

[0315] Real-time subtitle display

[0316] The captured text data is sent to the terminal and displayed on the user interface, allowing the user to read the subtitles displayed on the screen.

[0317] Sentiment analysis and interface adjustment

[0318] The device sends the user's facial expressions and tone of voice to an emotion engine, which then adjusts the display content and the tone and tempo of the audio playback based on the analysis results.

[0319] Example 2: For foreign users who do not understand Japanese

[0320] Voice input

[0321] A user is watching a variety show. The device captures the audio of the variety show and sends it to the server.

[0322] Text

[0323] The server sends the audio data from the variety show to the audio analysis API and obtains text data such as "This week's guest is a world-famous artist."

[0324] Voice conversion request

[0325] The user selects the subtitle portion, "This week's guest is a world-class artist," to hear it in English.

[0326] Voice generation

[0327] The server sends the selected text data to the speech generation API and obtains English speech data.

[0328] Audio playback and emotion analysis

[0329] The device plays back the acquired English audio data, while simultaneously analyzing the user's emotions using an emotion engine to adjust the tone and tempo of the audio playback.

[0330] The above is an example of a specific embodiment of the system according to the present invention, which aims to meet the diverse needs of users and make television viewing more accessible and personalized.

[0331] The processing flow will be explained below.

[0332] Step 1:

[0333] The device captures the audio of the TV program being watched by the user in real time using a built-in microphone or an external device, and stores the captured audio data in a buffer.

[0334] Step 2:

[0335] The terminal transmits the buffered audio data to the server at regular time intervals (for example, every second) using a protocol such as an HTTP POST request.

[0336] Step 3:

[0337] The server prepares to process the audio data received from the device and creates a request to send the audio data to the audio analysis API.

[0338] Step 4:

[0339] The server sends the voice data to the voice analysis API, which converts the voice data into text data.

[0340] Step 5:

[0341] The server receives the text data returned from the speech analysis API, formats the received text data, and adds timestamps, etc., as needed.

[0342] Step 6:

[0343] The server sends the formatted text data to the device using a WebSocket connection or an HTTP response.

[0344] Step 7:

[0345] The terminal acquires the text data received from the server and displays the acquired text data as real-time subtitles on the user interface.

[0346] Step 8:

[0347] The device captures the user's facial expressions or tone of voice in real time and sends them to the emotion engine, which analyzes the user's emotions and sends the results back to the device.

[0348] Step 9:

[0349] The device adjusts the displayed content of the text data and the tone and tempo of the voice based on the analysis results from the emotion engine. If the user's emotion is positive, the device brightens the displayed text and brightens the tone of the voice.

[0350] Step 10:

[0351] The user selects a specific portion of the displayed subtitles and clicks on the corresponding interface button when he or she wishes to hear the text spoken.

[0352] Step 11:

[0353] The terminal receives the user's selection operation and transmits the selected text data and a voice conversion request to the server.

[0354] Step 12:

[0355] The server creates a request to send the selected text data received from the device to the speech generation API.

[0356] Step 13:

[0357] The server sends the text data to the speech generation API and requests speech conversion, which converts the text data into speech data.

[0358] Step 14:

[0359] The server receives the voice data returned from the voice generation API and sends it to the device as is.

[0360] Step 15:

[0361] The terminal acquires the audio data received from the server and plays the acquired audio data using an audio player so that the user can hear it.

[0362] The above are the specific steps of the system's processing, and the specific operations at each step are described, allowing for a clear understanding of the system's operations.

[0363] Example 2

[0364] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0365] In conventional television broadcast viewing systems, there are users who cannot properly receive audio information due to hearing impairments or language barriers. For this reason, there is a need for a method to convert audio into text in real time and display that text to the user. However, existing systems have difficulty providing a personalized interface that takes user emotions into account. In addition, there is also the challenge of seamlessly capturing audio data in real time, converting it into text, and playing back the text as audio.

[0366] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a voice analysis means, a user interface means, a voice generation means, an emotion analysis means, and an interface adjustment means. This makes it possible to convert voice into text in real time and provide the text to the user in a personalized format. In addition, the process from capturing voice data to converting it into text and then playing the voice can be seamlessly linked, improving the user experience.

[0367] The "voice analysis means" is a means for analyzing voice data and converting it into text data.

[0368] The "user interface means" is a means for displaying text data to a user in real time and providing an interface for the user to input data.

[0369] The "voice generation means" is a means for analyzing text data and converting it into voice data.

[0370] The "emotion analysis means" is a means for capturing the user's facial expressions and tone of voice, and analyzing the data to recognize emotions.

[0371] The "interface adjustment means" is a means for adjusting the user interface, tone, tempo, etc. of audio playback based on the analyzed emotions.

[0372] System Overview

[0373] The present invention is a system that converts television broadcast audio into text in real time and displays the text on a user interface. It also has the function of converting user-selected text into speech and playing it back. The system can improve the user experience by recognizing the user's emotions and adjusting the interface based on them. This system is realized using a speech analysis means and a speech generation means.

[0374] Hardware and software used

[0375] Hardware:

[0376] Client device (terminal): Captures TV audio and displays text and plays audio.

[0377] Server: Analyzes and generates audio data.

[0378] software:

[0379] Speech analysis API: Analyzes voice data and converts it into text data (for example, Google Cloud Speech-to-Text API).

[0380] Speech generation API: Analyzes text data and converts it into speech data (e.g., Amazon Polly).

[0381] Emotion analysis API: Analyzes a user's facial expressions and tone of voice to recognize their emotions (for example, Microsoft Azure Cognitive Services Emotion API).

[0382] Program processing

[0383] 1. Capture and transmit audio

[0384] Terminal

[0385] While a user is watching a TV program, the device captures the TV audio in real time using a built-in microphone or a connected external device, and the captured audio data is periodically sent to a server.

[0386] 2. Speech-to-text

[0387] server

[0388] The server receives the voice data sent from the device and sends it to the voice analysis API. The voice analysis API analyzes the voice data and converts it into text data. The server then formats the converted text data and makes any necessary corrections to fit the context.

[0389] 3. Real-time subtitle generation and display

[0390] Terminal

[0391] Receives formatted text data sent from the server and displays it in the user interface in real time. Adjusts the font size and color of the text so that the user can easily read the text data.

[0392] 4. User's voice conversion request

[0393] User

[0394] The user selects a particular portion of text displayed on the screen and requests to hear the selected text aloud.

[0395] Terminal

[0396] It receives a user request and sends the text data to the server.

[0397] 5. Text-to-speech

[0398] server

[0399] The server sends the text data selected by the user to the speech generation API. The speech generation API analyzes the text data and converts it into speech data. The server receives the generated speech data and sends it back to the device.

[0400] 6. Playing audio data

[0401] Terminal

[0402] The device decompresses the received audio data and plays it through an audio player, providing an interface that allows you to adjust the volume and speed of the audio being played.

[0403] 7. Emotion Analysis Using an Emotion Engine

[0404] Terminal

[0405] The device captures the user's facial expressions or tone of voice using a built-in camera or microphone, and sends the captured data to the emotion engine, which analyzes the user's emotions in real time and sends the results back to the device.

[0406] 8. Emotion-Based Interface Adjustment

[0407] Terminal

[0408] Based on the analysis results from the emotion engine, the content displayed in the user interface and the tone and tempo of the audio data can be adjusted. For example, if the user shows a confused expression, the text font can be made larger and the audio playback speed can be slowed down.

[0409] Specific examples and prompts for the generative AI model

[0410] Example 1: A hearing-impaired user

[0411] Voice input

[0412] A user is watching a news program. During this time, the device captures the audio of the news and periodically transmits it to a server.

[0413] Text

[0414] The server sends the audio data of the news program to the audio analysis API and obtains the text data, "Tonight's news will report on important events from around the world."

[0415] Real-time subtitle display

[0416] The captured text data is sent to the terminal and displayed on the user interface, allowing the user to read the subtitles displayed on the screen.

[0417] Sentiment analysis and interface adjustment

[0418] The device sends the user's facial expressions and tone of voice to an emotion engine, which then adjusts the display content and the tone and tempo of the audio playback based on the analysis results.

[0419] Example 2: For foreign users who do not understand Japanese

[0420] Voice input

[0421] A user is watching a variety show. The device captures the audio of the variety show and sends it to the server.

[0422] Text

[0423] The server sends the audio data from the variety show to the audio analysis API and obtains text data such as "This week's guest is a world-famous artist."

[0424] Voice conversion request

[0425] The user selects the subtitle portion, "This week's guest is a world-class artist," to hear it in English.

[0426] Voice generation

[0427] The server sends the selected text data to the speech generation API and obtains English speech data.

[0428] Audio playback and emotion analysis

[0429] The device plays back the acquired English audio data, while simultaneously analyzing the user's emotions using an emotion engine to adjust the tone and tempo of the audio playback.

[0430] Example prompts for generative AI models

[0431] 1. Example prompts for transcription of news programs

[0432] "Capture the audio of news programs and convert it into text data in real time using speech analysis methods and display it."

[0433] 2. Example prompts for sentiment analysis

[0434] "Capture the user's facial expressions and tone of voice, use sentiment analysis tools to perform sentiment analysis, and adjust the interface based on the results."

[0435] The above is an example of a specific embodiment of the system according to the present invention, which aims to meet the diverse needs of users and make television viewing more accessible and personalized.

[0436] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0437] Step 1:

[0438] Audio capture and transmission

[0439] Terminal

[0440] While the user is watching a TV program, the device captures the TV audio in real time using the built-in microphone or a connected external device. This inputs the TV audio into the device as audio data. The captured audio data is temporarily stored in buffer memory. The audio data is then compressed at regular intervals (e.g., every 5 seconds) and sent to the server.

[0441] Step 2:

[0442] Speech to text

[0443] server

[0444] The server receives the voice data sent from the terminal. It then sends the received voice data to a voice analysis means (e.g., a voice analysis API). The voice analysis means analyzes the voice data and converts it into text data. During this process, the voice signal is digitized and context estimation is performed in conjunction with phonemic analysis. After the converted text data is sent back to the server, the server formats the text data and corrects it into an easy-to-read format. This formatted text data is then output.

[0445] Step 3:

[0446] Real-time subtitle generation and display

[0447] Terminal

[0448] The formatted text data sent from the server is received. The received text data is displayed in real time on the user interface. When displayed, the size and color of the characters, the transparency of the background, etc. are adjusted to make it easy for the user to read. At this time, the text data is input into the user interface and output in a visually represented form.

[0449] Step 4:

[0450] User's voice conversion request

[0451] User

[0452] The user selects a specific portion of text displayed on the screen. For example, they select a segment of a news program: "Tonight's news will report on important events from around the world." This selection is entered into the device's interface. The user then makes a request to hear the selected text aloud. This request triggers the transmission of the selected text data from the device to the server.

[0453] Step 5:

[0454] Text-to-speech

[0455] server

[0456] The server receives the text data selected by the user. This text data is sent to a speech generation means (e.g., a speech generation API). The speech generation means analyzes the text data and converts it into speech data. Specifically, it performs text analysis and generates a speech file using a speech synthesis engine. This generated speech data is sent back to the server, and then retransmitted from the server to the terminal.

[0457] Step 6:

[0458] Playing audio data

[0459] Terminal

[0460] The device decompresses the audio data received from the server and plays it through the built-in audio player. This audio data is played in an audio format (e.g., MP3) and output to the user's ears. The user has an interface to adjust the volume and speed of the audio being played.

[0461] Step 7:

[0462] Emotion analysis using an emotion engine

[0463] Terminal

[0464] The device uses a built-in camera and microphone to capture the user's facial expressions and tone of voice. This captured data is sent to an emotion analysis means (e.g., emotion analysis API). The emotion analysis means analyzes this data and identifies the user's emotions (e.g., joy, sadness, anger, confusion, etc.). The analysis results are sent back to the device in real time and output as the user's emotion data.

[0465] Step 8:

[0466] Emotion-based interface adjustment

[0467] Terminal

[0468] Based on the analysis results received from the emotion analysis means, the device adjusts the content displayed in the user interface and the tone and tempo of the audio data. For example, if the user shows a confused expression, the device increases the font size of the text and slows down the audio playback speed. This series of adjustment operations is reflected in the user interface and is output as an optimized interface.

[0469] This is the specific flow of the program processing for this system. The inputs and outputs at each processing step, as well as the specific operations, are clearly shown, allowing for a more detailed understanding of the operation of the entire system.

[0470] (Application example 2)

[0471] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0472] By converting television broadcast audio into text in real time and displaying it on a user interface, users with hearing impairments or difficulty understanding foreign languages ​​can easily understand the broadcast content. However, this system faces several challenges. For example, if a user selects a specific text section and wants to hear it aloud, fast and accurate speech conversion is required. Furthermore, to more individually optimize the user experience, it is necessary to analyze the user's emotions and state in real time and adjust the interface and playback speed accordingly.

[0473] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for capturing audio from a television broadcast, means for transmitting the captured audio data to a voice analysis API, means for displaying text data acquired from the voice analysis API on a user interface in real time, means for receiving a user's text selection and transmitting the selected text data to a voice generation API, means for playing back the audio data acquired from the voice generation API, means for capturing the user's facial expression or tone of voice and transmitting it to an emotion analysis engine, and means for adjusting the interface and playback speed based on the analysis results of the emotion analysis engine. This not only meets the diverse needs of users, but also makes it possible to optimize the user experience in a more personalized manner.

[0474] "Television broadcasting" is a broadcasting format that transmits images and sounds to ordinary households or specific regions using radio waves or cables.

[0475] "Audio capture" is the process of collecting audio data in digital form using a microphone or other audio input device.

[0476] "Audio data" is a digital representation of captured audio in a data format that can be analyzed and processed.

[0477] A "voice analysis API" is a programming interface for analyzing voice data and converting it into text data.

[0478] "Text data" is information stored in the form of character string data, and is used for displaying on a user interface or converting into voice.

[0479] "User interface" is a general term for the screens and input / output devices that allow users to interact with a system.

[0480] A "voice generation API" is a programming interface for converting text data into voice data.

[0481] "Audio data playback" is the process of audibly outputting digital audio data through an audio playback device.

[0482] An "emotion analysis engine" is a software component that analyzes a user's facial expressions and tone of voice to recognize their emotional state.

[0483] "Interface adjustment" is the process of changing the parameters of the user interface and audio playback based on the analysis results of the emotion analysis engine.

[0484] "Time-stamped text" refers to text data to which specific time information has been added, and corresponds to a specific portion of audio content.

[0485] MODE FOR CARRYING OUT THE INVENTION

[0486] Summary of the Invention Embodiments

[0487] The present invention relates to a system that converts television broadcast audio into text in real time and displays the text on a user interface. This system analyzes audio data captured during television viewing and generates text data. Furthermore, the system has a function that allows a user to select a specific text portion, converts the selected text into audio, and plays it back. Furthermore, by combining it with an emotion analysis engine, the system can adjust the interface and audio playback based on the user's emotions.

[0488] System Configuration

[0489] This system consists of three main components: a server, a terminal, and a user.

[0490] server:

[0491] Use a speech analysis API to convert audio data into text.

[0492] Uses the speech generation API to convert the selected text into audio data.

[0493] Device:

[0494] It captures audio data from television broadcasts in real time and displays the text.

[0495] It receives the user's voice requests and sends them to the server.

[0496] It uses a sentiment analysis engine to analyze user emotions and adjust the interface and audio playback accordingly.

[0497] User:

[0498] Watch a TV broadcast and select the displayed text to request a voice conversion.

[0499] Emotions are input through facial expressions and tone of voice.

[0500] Program processing

[0501] Audio capture and transmission

[0502] The device captures audio from a television broadcast using a built-in or external microphone and periodically transmits the captured audio to a server. For example, while watching a news program, the device continuously captures the audio of the news.

[0503] Speech to text

[0504] The server sends the captured audio data to a speech analysis API (such as the Google Speech API) and converts it into text data. This process generates text data such as "Tonight's news will report on important events from around the world."

[0505] Real-time text display

[0506] The generated text data is sent from the server to the terminal and displayed in real time on the user interface, allowing the user to visually confirm the news anchor's remarks as text in real time.

[0507] User's voice conversion request

[0508] The user selects a specific text segment and sends a request to the server via the terminal to listen to it aloud. For example, if the user selects "This week's guest is a world-famous artist," a voice conversion request for that segment is sent.

[0509] Text-to-speech

[0510] The server sends the selected text data to a speech generation API (e.g., Amazon Polly) to generate corresponding voice data, which is then sent back to the device.

[0511] Playing audio data

[0512] The device plays the received audio data through an audio player. In this case, after selecting the text "This week's guest is a world-famous artist," the user can listen to the content in English.

[0513] Analysis by sentiment analysis engine

[0514] The device captures the user's facial expressions and tone of voice and sends them to an emotion analysis engine (using OpenCV and FER, for example). Based on the analysis results, the device adjusts the user interface, audio playback speed, etc.

[0515] Prompt Sentence Examples

[0516] "Demonstrate an application that translates captured text in Japanese in real time and adjusts the interface and voice playback based on the user's emotions."

[0517] This makes it possible to provide a system that meets the diverse needs of users and makes television viewing more accessible and personalized.

[0518] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0519] Program processing flow

[0520] Step 1:

[0521] Audio capture and transmission

[0522] The device uses a built-in or external microphone to capture the audio of the TV broadcast in real time. It then periodically transmits the captured audio data to a server. The input is the audio of the TV broadcast, and the output is digital audio data. Specifically, the device's microphone is activated, the audio data is temporarily stored in a buffer, and the buffer is periodically uploaded to the server.

[0523] Step 2:

[0524] Speech to text

[0525] The server sends the received voice data to the voice analysis API. The voice analysis API converts the voice data into text data and sends the text data back to the server. The input is voice data and the output is text data. Specifically, the server calls the voice analysis API, sends the voice data, and receives the text data returned from the API.

[0526] Step 3:

[0527] Real-time text display

[0528] Text data sent from the server is sent to the terminal and displayed in real time on the user interface. The input is text data from the server, and the output is the text displayed on the user interface. Specifically, the terminal receives the text data and immediately reflects it in the display area of ​​the user interface.

[0529] Step 4:

[0530] User's voice conversion request

[0531] The user selects a specific portion of the displayed text and requests to hear the selected text aloud. The input is the selected text on the user interface, and the output is the requested data. Specific actions require the user to select the text using a touch screen or mouse and press a request button.

[0532] Step 5:

[0533] Text-to-speech

[0534] Text data requested from the device is sent to the server, which then uses a speech generation API to convert the text data into voice data. The voice data returned from the voice generation API is stored on the server and sent to the device. The input is the selected text data, and the output is voice data. Specifically, the server calls the voice generation API, sends the text data, and receives the generated voice data.

[0535] Step 6:

[0536] Playing audio data

[0537] The device plays the received audio data through an audio player. The input is the audio data from the server, and the output is the played audio. Specifically, the device starts the audio player, decodes the received audio data, and plays it.

[0538] Step 7:

[0539] Analysis by sentiment analysis engine

[0540] The device captures the user's facial expressions and tone of voice in real time and sends them to an emotion analysis engine. The emotion analysis engine analyzes the user's emotions and sends the results back to the device. The input is the captured video or audio data, and the output is the analyzed emotion data. Specific operations include activating the device's camera, capturing the user's facial expressions, and sending the video data to the emotion analysis engine.

[0541] Step 8:

[0542] Emotion-based interface adjustment

[0543] Based on the analysis results from the emotion analysis engine, the device adjusts the tone and tempo of the user interface and audio playback. The input is emotion analysis data, and the output is adjusted interface elements and audio playback. Specifically, the device automatically changes the color scheme of the user interface and adjusts the playback speed according to the analysis results.

[0544] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0545] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0546] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0547] [Second embodiment]

[0548] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0549] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0550] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0551] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0552] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0553] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0554] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0555] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0556] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0557] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0558] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0559] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0560] Summary of the Invention

[0561] This system converts television broadcast audio into text in real time and displays the text on a user interface. It also has the ability to convert user-selected text into speech and play it back. This system is implemented using a speech analysis API and a speech generation API.

[0562] System Configuration

[0563] This system consists of three main components: a server, a terminal, and a user. The server is responsible for converting speech to text and generating speech, while the terminal captures speech, displays text, and plays back speech. The user operates the system through the terminal.

[0564] Overview of program processing

[0565] 1. Capture and transmit audio

[0566] Terminal

[0567] While the user is watching a television program, the terminal captures the television sound in real time using a built-in microphone or a connected external device.

[0568] The captured audio data is periodically sent to a server.

[0569] 2. Speech-to-text

[0570] server

[0571] The server receives the voice data sent from the device and sends it to the voice analysis API.

[0572] The voice analysis API analyzes voice data and converts it into text data.

[0573] The server receives the converted text data and formats it.

[0574] 3. Real-time subtitle generation and display

[0575] Terminal

[0576] The formatted text data is sent to the terminal and displayed in real time on the user interface.

[0577] 4. User's voice conversion request

[0578] User

[0579] The user selects a particular portion of the displayed text and requests to hear that text aloud.

[0580] Terminal

[0581] This request is sent to the server along with the text data.

[0582] 5. Text-to-speech

[0583] server

[0584] The server sends the text data selected by the user to the speech generation API.

[0585] The speech generation API analyzes text data and converts it into speech data.

[0586] The server receives the generated audio data.

[0587] 6. Playing audio data

[0588] Terminal

[0589] The audio data is sent to the terminal and played back via an audio player.

[0590] Specific examples

[0591] Example 1: For a hearing impaired user

[0592] Voice input

[0593] A user is watching a news program. During this time, the device captures the audio of the news and periodically transmits it to a server.

[0594] Text

[0595] The server sends the audio data of the news program to a speech analysis API and obtains text data such as, "Tonight's news will report on important events from around the world."

[0596] Real-time subtitle display

[0597] The captured text data is sent to the terminal and displayed on the user interface, allowing the user to read the subtitles displayed on the screen.

[0598] Example 2: For foreign users who do not understand Japanese

[0599] Voice input

[0600] A user is watching a variety show. The device captures the audio of the variety show and sends it to the server.

[0601] Text

[0602] The server sends the audio data from the variety show to the audio analysis API and obtains text data such as "This week's guest is a world-famous artist."

[0603] Voice conversion request

[0604] The user selects the subtitle portion, "This week's guest is a world-class artist," to hear it in English.

[0605] Voice generation

[0606] The server sends the selected text data to the speech generation API and obtains English speech data.

[0607] Audio playback

[0608] The terminal plays back the acquired English audio data, allowing the user to understand the content.

[0609] The above is an example of a specific embodiment of the system according to the present invention, which aims to meet the diverse needs of users and make television viewing more accessible.

[0610] The processing flow will be explained below.

[0611] Step 1:

[0612] The device captures the audio of the TV program being watched by the user in real time using a built-in microphone or an external device, and stores the captured audio data in a buffer.

[0613] Step 2:

[0614] The terminal transmits the buffered audio data to the server at regular time intervals (for example, every second) using a protocol such as an HTTP POST request.

[0615] Step 3:

[0616] The server prepares to process the audio data received from the device and creates a request to send the audio data to the audio analysis API.

[0617] Step 4:

[0618] The server sends the voice data to the voice analysis API, which converts the voice data into text data.

[0619] Step 5:

[0620] The server receives the text data returned from the speech analysis API, formats the received text data, and adds timestamps, etc., as needed.

[0621] Step 6:

[0622] The server sends the formatted text data to the device using a WebSocket connection or an HTTP response.

[0623] Step 7:

[0624] The terminal acquires the text data received from the server and displays the acquired text data as real-time subtitles on the user interface.

[0625] Step 8:

[0626] The user selects a specific portion of the displayed subtitles and clicks on the corresponding interface button when he or she wishes to hear the text spoken.

[0627] Step 9:

[0628] The terminal receives the user's selection operation and transmits the selected text data and a voice conversion request to the server.

[0629] Step 10:

[0630] The server creates a request to send the selected text data received from the device to the speech generation API.

[0631] Step 11:

[0632] The server sends the text data to the speech generation API and requests speech conversion, which converts the text data into speech data.

[0633] Step 12:

[0634] The server receives the voice data returned from the voice generation API and sends it to the device as is.

[0635] Step 13:

[0636] The terminal acquires the audio data received from the server and plays the acquired audio data using an audio player so that the user can hear it.

[0637] The above are the specific steps of the system's processing, and the specific operations at each step are described, allowing the system's operations to be clearly understood.

[0638] Example 1

[0639] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0640] There is a need for a system that can convert television audio into text in real time and provide it to users in an easy-to-understand manner. In particular, users with hearing impairments and foreign users who speak different languages ​​need assistance in understanding television content. There is also a need for a system that allows users to select specific text and listen to it aloud.

[0641] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0642] In this invention, the server includes means for capturing audio data of a television broadcast, means for transmitting the captured audio data to the server, means for the server to transmit the audio data to an audio analysis API and format text data obtained from the audio analysis API, means for displaying the formatted text data on a user interface in real time, means for receiving a text selection from a user and transmitting the selected text data to an audio generation API, and means for playing the audio data obtained from the audio generation API, thereby enabling real-time conversion of audio to text and further playback of the user-selected text by audio.

[0643] "Means for capturing audio from a television broadcast" refers to means, including built-in microphones and external devices, used to collect audio from a television broadcast in real time.

[0644] The "means for transmitting captured audio data to a server" refers to a means having a function for periodically dividing collected audio data into batches and transmitting the batches to a server.

[0645] The "means for the server to transmit voice data to the voice analysis API" refers to a means including a process of transmitting voice data received by the server to the voice analysis API and converting the voice data into text data.

[0646] "Means for formatting text data obtained from a speech analysis API" refers to means for formatting text data obtained from a speech analysis API and for separating sentences and breaking lines to make the data easier to read.

[0647] "Means for displaying formatted text data in a user interface in real time" refers to means having the function of displaying formatted text data in a user interface in real time, including web pages and mobile applications.

[0648] The "means for receiving a user's text selection and transmitting the selected text data to the speech generation API" refers to a means that, when a user selects a specific portion of displayed text, transmits the selected text data to the speech generation API.

[0649] "Means for playing back audio data obtained from an audio generation API" refers to means for playing back audio data obtained from an audio generation API on a terminal, and includes an audio player.

[0650] MODE FOR CARRYING OUT THE INVENTION

[0651] overview

[0652] This system converts television broadcast audio into text in real time and displays the text on a user interface. It also has the ability to convert user-selected text into speech and play it back. This system is implemented using a speech analysis API and a speech generation API.

[0653] Major hardware and software used

[0654] Hardware:

[0655] Device: Built-in microphone or external device (e.g. USB microphone)

[0656] software:

[0657] Speech analysis API: For example, Google Cloud Speech-to-Text API

[0658] Speech generation API: For example, Amazon Polly

[0659] User interface: HTML, JavaScript, and a front-end framework such as React or Vue.js

[0660] Example of operation

[0661] Audio capture and transmission

[0662] Terminal

[0663] While a user is watching a TV program, the device captures the TV audio in real time through a built-in microphone or an external device. Specifically, an audio capture application on the device monitors the microphone input and captures audio data in real time.

[0664] The captured audio data is periodically split into small batches and sent to the server, for example every 5 seconds.

[0665] Speech to text

[0666] server

[0667] The server receives the voice data sent from the device via an HTTP request, and sends the received voice data to a voice analysis API (e.g., Google Cloud Speech-to-Text API).

[0668] The speech analysis API converts the speech data into text data, and the server receives the converted text data and performs any necessary formatting, such as adding sentence breaks and line breaks to make the text easier to read.

[0669] Real-time subtitle display

[0670] Terminal

[0671] The formatted text data is sent from the server to the terminal. Specifically, the server sends the text data as an HTTP response.

[0672] The device displays this text data in a user interface (e.g., web page, mobile app) in real time. Using JavaScript or a front-end framework, the text can be displayed dynamically.

[0673] User's voice conversion request

[0674] User

[0675] The user selects a particular portion of the displayed text, for example by clicking on a block of text.

[0676] Terminal

[0677] Based on this selection, the text data and a request for speech conversion are sent to the server. Specifically, the selected text and the request type (e.g., speech conversion request) are sent to the server as a POST request.

[0678] Text-to-speech

[0679] server

[0680] The server receives the text data selected by the user and extracts it from the JSON body of the HTTP request.

[0681] Send the received text data to a speech generation API (e.g., Amazon Polly), authenticating with an API key and specifying the required parameters (e.g., language, tone of voice).

[0682] The speech generation API analyzes the text data and converts it into speech data, and the server receives the generated speech data.

[0683] Playing audio data

[0684] Terminal

[0685] The generated voice data is sent from the server to the terminal, usually as an HTTP response.

[0686] The device receives this audio data and plays it using an audio player (e.g. HTML5 Audio). JavaScript is used to set the URL of the audio data in the audio player, and the audio is played when the play button is pressed.

[0687] Examples of prompt statements

[0688] "Please explain in natural language a system that converts audio from a news program into text and plays back the text selected by the user as speech."

[0689] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0690] Step 1: Capture and send audio

[0691] Terminal

[0692] While the user is watching a TV program, the device captures TV audio in real time using a built-in microphone or an external device. The captured audio data is divided into batches every 5 seconds. The device periodically sends these audio data batches to the server. The input is the captured audio data, and the output is the batched audio data. This capture process is performed using the Audio Capture API, and transmission to the server is via an HTTP POST request.

[0693] Step 2: Speech to text

[0694] server

[0695] The server receives the audio data batches sent from the device. It sends the received audio data batches to a speech analysis API and converts the audio data into text data. Here, the Google Cloud Speech-to-Text API is used. The converted text data is received and formatted. Specifically, it adds sentence separators and line breaks to make it easier to read. The input is the audio data batches, and the output is formatted text data.

[0696] Step 3: Display real-time subtitles

[0697] Terminal

[0698] The terminal receives formatted text data sent from the server. The terminal displays this text data in the user interface in real time. The text is displayed dynamically using JavaScript and a front-end framework (e.g., React, Vue.js). The input is formatted text data, and the output is real-time subtitles displayed in the user interface.

[0699] Step 4: User's voice conversion request

[0700] User

[0701] The user selects a specific portion of the displayed text, for example by clicking on a block of text. The input is the text data displayed in real time, and the output is the text data selected by the user.

[0702] Step 5: Sending the selected text

[0703] Terminal

[0704] It receives the text data selected by the user and sends it to the server. Specifically, it sends the selected text data and the speech conversion request information to the server as a POST request. The input is the text data selected by the user, and the output is an HTTP request to the server.

[0705] Step 6: Convert text to speech

[0706] server

[0707] The server sends the received selected text data to the speech generation API. Amazon Polly is used here. Authentication is performed using an API key, and a speech conversion request is sent specifying the required parameters (e.g., language, tone of voice). The server receives the speech data obtained from the speech generation API. The input is the selected text data, and the output is the generated speech data.

[0708] Step 7: Playing back audio data

[0709] Terminal

[0710] Receives audio data sent from the server. The device receives the audio data and plays it using an audio player (e.g. HTML5 Audio). JavaScript is used to set the URL of the audio data in the audio player, and the audio is played when the play button is pressed. The input is the received audio data, and the output is the audio played to the user.

[0711] (Application example 1)

[0712] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0713] Conventional food delivery applications lack effective means of conveying audio information, such as restaurant promotional videos and menu introductions, to users. This poses a challenge, making it difficult to provide sufficient information to users with hearing impairments or who have difficulty understanding foreign languages. Furthermore, the lack of information provision through audiovisual interaction raises concerns about a decline in the quality of the user experience.

[0714] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0715] In this invention, the server includes means for transmitting voice data to a voice analysis API and acquiring text data, means for formatting the acquired text data and assigning a timestamp, means for displaying the text data on a user interface, means for receiving a user's text selection and transmitting it to a voice generation API to acquire voice data, means for playing the voice data, and means for capturing and converting the voice of an advertising medium into text to provide food delivery information, and converting the text selected by the user into voice and playing it back. This allows the user to visually check the voice information converted into text in real time, and further select the text as needed and play it back as voice information.

[0716] "Information media" refers to means of providing information through audio and video. This includes television broadcasts, radio broadcasts, online videos, etc.

[0717] A "voice analysis API" is an application programming interface (API) for analyzing voice data and converting it into text data. It is generally implemented using a cloud-based service.

[0718] A "user interface" is an interface that allows a user to interact with a system. It includes means such as a screen display and audio output.

[0719] A "Speech Generation API" is an application programming interface (API) for converting text data into speech data. It provides text-to-speech functionality.

[0720] "Audio data" means a digital representation of sound waves, typically captured by a microphone or recorded audio.

[0721] "Text data" refers to data written in characters. This includes text generated from audio data using a voice analysis API.

[0722] "Food delivery information" refers to information about food and drink delivery, including restaurant promotional videos, menu introductions, delivery status, etc.

[0723] "Promotional media" refers to the means of widely disseminating product information. This includes videos, audio, posters, etc.

[0724] This invention is a system that converts audio from television broadcasts and other information media into text in real time and displays it on a user interface. It also has the function of converting user-selected text into audio and playing it back. The following are specific examples of how this system can be implemented.

[0725] System Configuration

[0726] The system consists of three main components: a server, a terminal, and a user.

[0727] server

[0728] The server operates a program that uses the voice analysis API and voice generation API, which provides the following functions:

[0729] Send the audio data to the audio analysis API and obtain the text data.

[0730] The acquired text data is formatted and a timestamp is added.

[0731] The formatted text data is sent to the terminal and displayed.

[0732] It receives a text selection request from the user, sends the text to the speech generation API, and obtains the speech data.

[0733] Terminal

[0734] A terminal is a device (e.g., a smartphone or tablet) that is operated by a user. A terminal performs the following functions:

[0735] Capture audio using the built-in microphone or an external device.

[0736] Send the captured audio data to the server.

[0737] The text data transmitted from the server is displayed on the user interface in real time.

[0738] The text data selected by the user is sent again to the server, and the voice data is acquired.

[0739] The acquired audio data is played back.

[0740] User

[0741] The User is the final user of the system and performs the following operations:

[0742] Watch television broadcasts and other information media and capture audio through your device.

[0743] View real-time text data displayed in the user interface.

[0744] Select a specific portion of text and make a speech conversion request.

[0745] Listen to the audio being played and understand the information.

[0746] Program processing procedure

[0747] The server first receives the voice data sent from the device. Next, it sends the voice data to a voice analysis API to obtain text data. This text data is formatted and a timestamp is added. The formatted text is sent to the device and displayed in the user interface. When the user selects a portion of the text and requests voice conversion, the server sends the text data to a voice generation API to obtain the voice data. Finally, the voice data is sent to the device and played.

[0748] Hardware and software used

[0749] Hardware

[0750] Smartphones, tablets, and microphones

[0751] software

[0752] Speech analysis API (e.g., Google Cloud Speech-to-Text)

[0753] Speech generation APIs (e.g., Google Text-to-Speech)

[0754] Server-side program (implemented in Python etc.)

[0755] User Interface (using a front-end framework)

[0756] Specific examples

[0757] When a user uses a food delivery app, the audio from a restaurant's promotional video is captured by the smartphone's microphone. This audio is converted into text in real time and displayed on the user interface. The user selects the part of the displayed text that says "Today's special is sushi pizza" and requests that it be played aloud. The server sends this text to a speech generation API, which then transmits the generated audio to the device. Finally, the user hears the audio saying "sushi pizza."

[0758] Prompt Sentence Examples

[0759] "When a user types "sushi pizza" into your application, convert the text into speech and play it back."

[0760] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0761] Output results according to processing steps

[0762] Step 1:

[0763] The device captures audio from information media using a built-in microphone or an external device, and this audio data is collected in real time and sent to a server for further processing.

[0764] Input: Audio data (audio from information media)

[0765] Output: Audio data is sent to the server

[0766] Step 2:

[0767] The server receives the voice data sent from the device, then uses a voice analysis API to convert the voice data into text data, which is then formatted and timestamped.

[0768] Input: Audio data

[0769] Output: Formatted text data (with timestamp)

[0770] Step 3:

[0771] The server sends the formatted text data to the terminal.

[0772] Input: Formatted text data

[0773] Output: Text data is sent to the terminal

[0774] Step 4:

[0775] The terminal displays the transmitted text data on the user interface in real time, allowing the user to check the text data displayed in a visible format.

[0776] Input: Text data

[0777] Output: Real-time displayed text data

[0778] Step 5:

[0779] The user selects the desired part from the text data, and the selected text is sent back to the server as a speech generation request.

[0780] Input: Real-time displayed text data

[0781] Output: The selected text data is sent to the server.

[0782] Step 6:

[0783] The server uses a speech generation API to convert the selected text data into speech data, which is then sent to the device.

[0784] Input: Selected text data

[0785] Output: Generated audio data

[0786] Step 7:

[0787] The terminal plays the audio data that has been sent, and the user can hear the audio corresponding to the selected text.

[0788] Input: Generated audio data

[0789] Output: Played audio data (user hears the audio)

[0790] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0791] Summary of the Invention

[0792] This invention is a system that converts television broadcast audio into text in real time and displays that text on a user interface. It also has the ability to convert user-selected text into speech and play it back. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, the user experience can be further improved. This system is realized using a speech analysis API and a speech generation API.

[0793] System Configuration

[0794] This system consists of three main components: a server, a terminal, and a user. The server is responsible for converting speech to text and generating speech, while the terminal is responsible for capturing speech, displaying text, playing speech, and analyzing emotions using an emotion engine. The user operates the system through the terminal.

[0795] Overview of program processing

[0796] 1. Capture and transmit audio

[0797] Terminal

[0798] While the user is watching a television program, the terminal captures the television sound in real time using a built-in microphone or a connected external device.

[0799] The captured audio data is periodically sent to a server.

[0800] 2. Speech-to-text

[0801] server

[0802] The server receives the voice data sent from the device and sends it to the voice analysis API.

[0803] The voice analysis API analyzes voice data and converts it into text data.

[0804] The server receives the converted text data and formats it.

[0805] 3. Real-time subtitle generation and display

[0806] Terminal

[0807] The formatted text data is sent to the terminal and displayed in real time on the user interface.

[0808] 4. User's voice conversion request

[0809] User

[0810] The user selects a particular portion of the displayed text and requests to hear that text aloud.

[0811] Terminal

[0812] This request is sent to the server along with the text data.

[0813] 5. Text-to-speech

[0814] server

[0815] The server sends the text data selected by the user to the speech generation API.

[0816] The speech generation API analyzes text data and converts it into speech data.

[0817] The server receives the generated audio data.

[0818] 6. Playing audio data

[0819] Terminal

[0820] The audio data is sent to the terminal and played back via an audio player.

[0821] 7. Emotion Analysis Using an Emotion Engine

[0822] Terminal

[0823] The device captures the user's facial expressions or tone of voice in real time and sends them to the emotion engine.

[0824] The emotion engine analyzes the user's emotions and returns the results to the terminal.

[0825] 8. Emotion-Based Interface Adjustment

[0826] Terminal

[0827] Based on the analysis results from the emotion engine, the content of the text data displayed and the tone and tempo of the audio data are adjusted.

[0828] Specific examples

[0829] Example 1: For a hearing impaired user

[0830] Voice input

[0831] A user is watching a news program. During this time, the device captures the audio of the news and periodically transmits it to a server.

[0832] Text

[0833] The server sends the audio data of the news program to a speech analysis API and obtains text data such as, "Tonight's news will report on important events from around the world."

[0834] Real-time subtitle display

[0835] The captured text data is sent to the terminal and displayed on the user interface, allowing the user to read the subtitles displayed on the screen.

[0836] Sentiment analysis and interface adjustment

[0837] The device sends the user's facial expressions and tone of voice to an emotion engine, which then adjusts the display content and the tone and tempo of the audio playback based on the analysis results.

[0838] Example 2: For foreign users who do not understand Japanese

[0839] Voice input

[0840] A user is watching a variety show. The device captures the audio of the variety show and sends it to the server.

[0841] Text

[0842] The server sends the audio data from the variety show to the audio analysis API and obtains text data such as "This week's guest is a world-famous artist."

[0843] Voice conversion request

[0844] The user selects the subtitle portion, "This week's guest is a world-class artist," to hear it in English.

[0845] Voice generation

[0846] The server sends the selected text data to the speech generation API and obtains English speech data.

[0847] Audio playback and emotion analysis

[0848] The device plays back the acquired English audio data, while simultaneously analyzing the user's emotions using an emotion engine to adjust the tone and tempo of the audio playback.

[0849] The above is an example of a specific embodiment of the system according to the present invention, which aims to meet the diverse needs of users and make television viewing more accessible and personalized.

[0850] The processing flow will be explained below.

[0851] Step 1:

[0852] The device captures the audio of the TV program being watched by the user in real time using a built-in microphone or an external device, and stores the captured audio data in a buffer.

[0853] Step 2:

[0854] The terminal transmits the buffered audio data to the server at regular time intervals (for example, every second) using a protocol such as an HTTP POST request.

[0855] Step 3:

[0856] The server prepares to process the audio data received from the device and creates a request to send the audio data to the audio analysis API.

[0857] Step 4:

[0858] The server sends the voice data to the voice analysis API, which converts the voice data into text data.

[0859] Step 5:

[0860] The server receives the text data returned from the speech analysis API, formats the received text data, and adds timestamps, etc., as needed.

[0861] Step 6:

[0862] The server sends the formatted text data to the device using a WebSocket connection or an HTTP response.

[0863] Step 7:

[0864] The terminal acquires the text data received from the server and displays the acquired text data as real-time subtitles on the user interface.

[0865] Step 8:

[0866] The device captures the user's facial expressions or tone of voice in real time and sends them to the emotion engine, which analyzes the user's emotions and sends the results back to the device.

[0867] Step 9:

[0868] The device adjusts the displayed content of the text data and the tone and tempo of the voice based on the analysis results from the emotion engine. If the user's emotion is positive, the device brightens the displayed text and brightens the tone of the voice.

[0869] Step 10:

[0870] The user selects a specific portion of the displayed subtitles and clicks on the corresponding interface button when he or she wishes to hear the text spoken.

[0871] Step 11:

[0872] The terminal receives the user's selection operation and transmits the selected text data and a voice conversion request to the server.

[0873] Step 12:

[0874] The server creates a request to send the selected text data received from the device to the speech generation API.

[0875] Step 13:

[0876] The server sends the text data to the speech generation API and requests speech conversion, which converts the text data into speech data.

[0877] Step 14:

[0878] The server receives the voice data returned from the voice generation API and sends it to the device as is.

[0879] Step 15:

[0880] The terminal acquires the audio data received from the server and plays the acquired audio data using an audio player so that the user can hear it.

[0881] The above are the specific steps of the system's processing, and the specific operations at each step are described, allowing for a clear understanding of the system's operations.

[0882] Example 2

[0883] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0884] In conventional television broadcast viewing systems, there are users who cannot properly receive audio information due to hearing impairments or language barriers. For this reason, there is a need for a method to convert audio into text in real time and display that text to the user. However, existing systems have difficulty providing a personalized interface that takes user emotions into account. In addition, there is also the challenge of seamlessly capturing audio data in real time, converting it into text, and playing back the text as audio.

[0885] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a voice analysis means, a user interface means, a voice generation means, an emotion analysis means, and an interface adjustment means. This makes it possible to convert voice into text in real time and provide the text to the user in a personalized format. In addition, the process from capturing voice data to converting it into text and then playing the voice can be seamlessly linked, improving the user experience.

[0886] The "voice analysis means" is a means for analyzing voice data and converting it into text data.

[0887] The "user interface means" is a means for displaying text data to a user in real time and providing an interface for the user to input data.

[0888] The "voice generation means" is a means for analyzing text data and converting it into voice data.

[0889] The "emotion analysis means" is a means for capturing the user's facial expressions and tone of voice, and analyzing the data to recognize emotions.

[0890] The "interface adjustment means" is a means for adjusting the user interface, tone, tempo, etc. of audio playback based on the analyzed emotions.

[0891] System Overview

[0892] The present invention is a system that converts television broadcast audio into text in real time and displays the text on a user interface. It also has the function of converting user-selected text into speech and playing it back. The system can improve the user experience by recognizing the user's emotions and adjusting the interface based on them. This system is realized using a speech analysis means and a speech generation means.

[0893] Hardware and software used

[0894] Hardware:

[0895] Client device (terminal): Captures TV audio and displays text and plays audio.

[0896] Server: Analyzes and generates audio data.

[0897] software:

[0898] Speech analysis API: Analyzes voice data and converts it into text data (for example, Google Cloud Speech-to-Text API).

[0899] Speech generation API: Analyzes text data and converts it into speech data (e.g., Amazon Polly).

[0900] Emotion analysis API: Analyzes a user's facial expressions and tone of voice to recognize their emotions (for example, Microsoft Azure Cognitive Services Emotion API).

[0901] Program processing

[0902] 1. Capture and transmit audio

[0903] Terminal

[0904] While a user is watching a TV program, the device captures the TV audio in real time using a built-in microphone or a connected external device, and the captured audio data is periodically sent to a server.

[0905] 2. Speech-to-text

[0906] server

[0907] The server receives the voice data sent from the device and sends it to the voice analysis API. The voice analysis API analyzes the voice data and converts it into text data. The server then formats the converted text data and makes any necessary corrections to fit the context.

[0908] 3. Real-time subtitle generation and display

[0909] Terminal

[0910] Receives formatted text data sent from the server and displays it in the user interface in real time. Adjusts the font size and color of the text so that the user can easily read the text data.

[0911] 4. User's voice conversion request

[0912] User

[0913] The user selects a particular portion of text displayed on the screen and requests to hear the selected text aloud.

[0914] Terminal

[0915] It receives a user request and sends the text data to the server.

[0916] 5. Text-to-speech

[0917] server

[0918] The server sends the text data selected by the user to the speech generation API. The speech generation API analyzes the text data and converts it into speech data. The server receives the generated speech data and sends it back to the device.

[0919] 6. Playing audio data

[0920] Terminal

[0921] The device decompresses the received audio data and plays it through an audio player, providing an interface that allows you to adjust the volume and speed of the audio being played.

[0922] 7. Emotion Analysis Using an Emotion Engine

[0923] Terminal

[0924] The device captures the user's facial expressions or tone of voice using a built-in camera or microphone, and sends the captured data to the emotion engine, which analyzes the user's emotions in real time and sends the results back to the device.

[0925] 8. Emotion-Based Interface Adjustment

[0926] Terminal

[0927] Based on the analysis results from the emotion engine, the content displayed in the user interface and the tone and tempo of the audio data can be adjusted. For example, if the user shows a confused expression, the text font can be made larger and the audio playback speed can be slowed down.

[0928] Specific examples and prompts for the generative AI model

[0929] Example 1: A hearing-impaired user

[0930] Voice input

[0931] A user is watching a news program. During this time, the device captures the audio of the news and periodically transmits it to a server.

[0932] Text

[0933] The server sends the audio data of the news program to the audio analysis API and obtains the text data, "Tonight's news will report on important events from around the world."

[0934] Real-time subtitle display

[0935] The captured text data is sent to the terminal and displayed on the user interface, allowing the user to read the subtitles displayed on the screen.

[0936] Sentiment analysis and interface adjustment

[0937] The device sends the user's facial expressions and tone of voice to an emotion engine, which then adjusts the display content and the tone and tempo of the audio playback based on the analysis results.

[0938] Example 2: For foreign users who do not understand Japanese

[0939] Voice input

[0940] A user is watching a variety show. The device captures the audio of the variety show and sends it to the server.

[0941] Text

[0942] The server sends the audio data from the variety show to the audio analysis API and obtains text data such as "This week's guest is a world-famous artist."

[0943] Voice conversion request

[0944] The user selects the subtitle portion, "This week's guest is a world-class artist," to hear it in English.

[0945] Voice generation

[0946] The server sends the selected text data to the speech generation API and obtains English speech data.

[0947] Audio playback and emotion analysis

[0948] The device plays back the acquired English audio data, while simultaneously analyzing the user's emotions using an emotion engine to adjust the tone and tempo of the audio playback.

[0949] Example prompts for generative AI models

[0950] 1. Example prompts for transcription of news programs

[0951] "Capture the audio of news programs and convert it into text data in real time using speech analysis methods and display it."

[0952] 2. Example prompts for sentiment analysis

[0953] "Capture the user's facial expressions and tone of voice, use sentiment analysis tools to perform sentiment analysis, and adjust the interface based on the results."

[0954] The above is an example of a specific embodiment of the system according to the present invention, which aims to meet the diverse needs of users and make television viewing more accessible and personalized.

[0955] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0956] Step 1:

[0957] Audio capture and transmission

[0958] Terminal

[0959] While the user is watching a TV program, the device captures the TV audio in real time using the built-in microphone or a connected external device. This inputs the TV audio into the device as audio data. The captured audio data is temporarily stored in buffer memory. The audio data is then compressed at regular intervals (e.g., every 5 seconds) and sent to the server.

[0960] Step 2:

[0961] Speech to text

[0962] server

[0963] The server receives the voice data sent from the terminal. It then sends the received voice data to a voice analysis means (e.g., a voice analysis API). The voice analysis means analyzes the voice data and converts it into text data. During this process, the voice signal is digitized and context estimation is performed in conjunction with phonemic analysis. After the converted text data is sent back to the server, the server formats the text data and corrects it into an easy-to-read format. This formatted text data is then output.

[0964] Step 3:

[0965] Real-time subtitle generation and display

[0966] Terminal

[0967] The formatted text data sent from the server is received. The received text data is displayed in real time on the user interface. When displayed, the size and color of the characters, the transparency of the background, etc. are adjusted to make it easy for the user to read. At this time, the text data is input into the user interface and output in a visually represented form.

[0968] Step 4:

[0969] User's voice conversion request

[0970] User

[0971] The user selects a specific portion of text displayed on the screen. For example, they select a segment of a news program: "Tonight's news will report on important events from around the world." This selection is entered into the device's interface. The user then makes a request to hear the selected text aloud. This request triggers the transmission of the selected text data from the device to the server.

[0972] Step 5:

[0973] Text-to-speech

[0974] server

[0975] The server receives the text data selected by the user. This text data is sent to a speech generation means (e.g., a speech generation API). The speech generation means analyzes the text data and converts it into speech data. Specifically, it performs text analysis and generates a speech file using a speech synthesis engine. This generated speech data is sent back to the server, and then retransmitted from the server to the terminal.

[0976] Step 6:

[0977] Playing audio data

[0978] Terminal

[0979] The device decompresses the audio data received from the server and plays it through the built-in audio player. This audio data is played in an audio format (e.g., MP3) and output to the user's ears. The user has an interface to adjust the volume and speed of the audio being played.

[0980] Step 7:

[0981] Emotion analysis using an emotion engine

[0982] Terminal

[0983] The device uses a built-in camera and microphone to capture the user's facial expressions and tone of voice. This captured data is sent to an emotion analysis means (e.g., emotion analysis API). The emotion analysis means analyzes this data and identifies the user's emotions (e.g., joy, sadness, anger, confusion, etc.). The analysis results are sent back to the device in real time and output as the user's emotion data.

[0984] Step 8:

[0985] Emotion-based interface adjustment

[0986] Terminal

[0987] Based on the analysis results received from the emotion analysis means, the device adjusts the content displayed in the user interface and the tone and tempo of the audio data. For example, if the user shows a confused expression, the device increases the font size of the text and slows down the audio playback speed. This series of adjustment operations is reflected in the user interface and is output as an optimized interface.

[0988] This is the specific flow of the program processing for this system. The inputs and outputs at each processing step, as well as the specific operations, are clearly shown, allowing for a more detailed understanding of the operation of the entire system.

[0989] (Application example 2)

[0990] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0991] By converting television broadcast audio into text in real time and displaying it on a user interface, users with hearing impairments or difficulty understanding foreign languages ​​can easily understand the broadcast content. However, this system faces several challenges. For example, if a user selects a specific text section and wants to hear it aloud, fast and accurate speech conversion is required. Furthermore, to more individually optimize the user experience, it is necessary to analyze the user's emotions and state in real time and adjust the interface and playback speed accordingly.

[0992] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for capturing audio from a television broadcast, means for transmitting the captured audio data to a voice analysis API, means for displaying text data acquired from the voice analysis API on a user interface in real time, means for receiving a user's text selection and transmitting the selected text data to a voice generation API, means for playing back the audio data acquired from the voice generation API, means for capturing the user's facial expression or tone of voice and transmitting it to an emotion analysis engine, and means for adjusting the interface and playback speed based on the analysis results of the emotion analysis engine. This not only meets the diverse needs of users, but also makes it possible to optimize the user experience in a more personalized manner.

[0993] "Television broadcasting" is a broadcasting format that transmits images and sounds to ordinary households or specific regions using radio waves or cables.

[0994] "Audio capture" is the process of collecting audio data in digital form using a microphone or other audio input device.

[0995] "Audio data" is a digital representation of captured audio in a data format that can be analyzed and processed.

[0996] A "voice analysis API" is a programming interface for analyzing voice data and converting it into text data.

[0997] "Text data" is information stored in the form of character string data, and is used for displaying on a user interface or converting into voice.

[0998] "User interface" is a general term for the screens and input / output devices that allow users to interact with a system.

[0999] A "voice generation API" is a programming interface for converting text data into voice data.

[1000] "Audio data playback" is the process of audibly outputting digital audio data through an audio playback device.

[1001] An "emotion analysis engine" is a software component that analyzes a user's facial expressions and tone of voice to recognize their emotional state.

[1002] "Interface adjustment" is the process of changing the parameters of the user interface and audio playback based on the analysis results of the emotion analysis engine.

[1003] "Time-stamped text" refers to text data to which specific time information has been added, and corresponds to a specific portion of audio content.

[1004] MODE FOR CARRYING OUT THE INVENTION

[1005] Summary of the Invention Embodiments

[1006] The present invention relates to a system that converts television broadcast audio into text in real time and displays the text on a user interface. This system analyzes audio data captured during television viewing and generates text data. Furthermore, the system has a function that allows a user to select a specific text portion, converts the selected text into audio, and plays it back. Furthermore, by combining it with an emotion analysis engine, the system can adjust the interface and audio playback based on the user's emotions.

[1007] System Configuration

[1008] This system consists of three main components: a server, a terminal, and a user.

[1009] server:

[1010] Use a speech analysis API to convert audio data into text.

[1011] Uses the speech generation API to convert the selected text into audio data.

[1012] Device:

[1013] It captures audio data from television broadcasts in real time and displays the text.

[1014] It receives the user's voice requests and sends them to the server.

[1015] It uses a sentiment analysis engine to analyze user emotions and adjust the interface and audio playback accordingly.

[1016] User:

[1017] Watch a TV broadcast and select the displayed text to request a voice conversion.

[1018] Emotions are input through facial expressions and tone of voice.

[1019] Program processing

[1020] Audio capture and transmission

[1021] The device captures audio from a television broadcast using a built-in or external microphone and periodically transmits the captured audio to a server. For example, while watching a news program, the device continuously captures the audio of the news.

[1022] Speech to text

[1023] The server sends the captured audio data to a speech analysis API (such as the Google Speech API) and converts it into text data. This process generates text data such as "Tonight's news will report on important events from around the world."

[1024] Real-time text display

[1025] The generated text data is sent from the server to the terminal and displayed in real time on the user interface, allowing the user to visually confirm the news anchor's remarks as text in real time.

[1026] User's voice conversion request

[1027] The user selects a specific text segment and sends a request to the server via the terminal to listen to it aloud. For example, if the user selects "This week's guest is a world-famous artist," a voice conversion request for that segment is sent.

[1028] Text-to-speech

[1029] The server sends the selected text data to a speech generation API (e.g., Amazon Polly) to generate corresponding voice data, which is then sent back to the device.

[1030] Playing audio data

[1031] The device plays the received audio data through an audio player. In this case, after selecting the text "This week's guest is a world-famous artist," the user can listen to the content in English.

[1032] Analysis by sentiment analysis engine

[1033] The device captures the user's facial expressions and tone of voice and sends them to an emotion analysis engine (using OpenCV and FER, for example). Based on the analysis results, the device adjusts the user interface, audio playback speed, etc.

[1034] Prompt Sentence Examples

[1035] "Demonstrate an application that translates captured text in Japanese in real time and adjusts the interface and audio playback based on the user's emotions."

[1036] This makes it possible to provide a system that meets the diverse needs of users and makes television viewing more accessible and personalized.

[1037] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1038] Program processing flow

[1039] Step 1:

[1040] Audio capture and transmission

[1041] The device uses a built-in or external microphone to capture the audio of the TV broadcast in real time. It then periodically transmits the captured audio data to a server. The input is the audio of the TV broadcast, and the output is digital audio data. Specifically, the device's microphone is activated, the audio data is temporarily stored in a buffer, and the buffer is periodically uploaded to the server.

[1042] Step 2:

[1043] Speech to text

[1044] The server sends the received voice data to the voice analysis API. The voice analysis API converts the voice data into text data and sends the text data back to the server. The input is voice data and the output is text data. Specifically, the server calls the voice analysis API, sends the voice data, and receives the text data returned from the API.

[1045] Step 3:

[1046] Real-time text display

[1047] Text data sent from the server is sent to the terminal and displayed in real time on the user interface. The input is text data from the server, and the output is the text displayed on the user interface. Specifically, the terminal receives the text data and immediately reflects it in the display area of ​​the user interface.

[1048] Step 4:

[1049] User's voice conversion request

[1050] The user selects a specific portion of the displayed text and requests to hear the selected text aloud. The input is the selected text on the user interface, and the output is the requested data. Specific actions require the user to select the text using a touch screen or mouse and press a request button.

[1051] Step 5:

[1052] Text-to-speech

[1053] Text data requested from the device is sent to the server, which then uses a speech generation API to convert the text data into voice data. The voice data returned from the voice generation API is stored on the server and sent to the device. The input is the selected text data, and the output is voice data. Specifically, the server calls the voice generation API, sends the text data, and receives the generated voice data.

[1054] Step 6:

[1055] Playing audio data

[1056] The device plays the received audio data through an audio player. The input is the audio data from the server, and the output is the played audio. Specifically, the device starts the audio player, decodes the received audio data, and plays it.

[1057] Step 7:

[1058] Analysis by sentiment analysis engine

[1059] The device captures the user's facial expressions and tone of voice in real time and sends them to an emotion analysis engine. The emotion analysis engine analyzes the user's emotions and sends the results back to the device. The input is the captured video or audio data, and the output is the analyzed emotion data. Specific operations include activating the device's camera, capturing the user's facial expressions, and sending the video data to the emotion analysis engine.

[1060] Step 8:

[1061] Emotion-based interface adjustment

[1062] Based on the analysis results from the emotion analysis engine, the device adjusts the tone and tempo of the user interface and audio playback. The input is emotion analysis data, and the output is adjusted interface elements and audio playback. Specifically, the device automatically changes the color scheme of the user interface and adjusts the playback speed according to the analysis results.

[1063] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1064] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1065] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1066] [Third embodiment]

[1067] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1068] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1069] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1070] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1071] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1072] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1073] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1074] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1075] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1076] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1077] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1078] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1079] Summary of the Invention

[1080] This system converts television broadcast audio into text in real time and displays the text on a user interface. It also has the ability to convert user-selected text into speech and play it back. This system is implemented using a speech analysis API and a speech generation API.

[1081] System Configuration

[1082] This system consists of three main components: a server, a terminal, and a user. The server is responsible for converting speech to text and generating speech, while the terminal captures speech, displays text, and plays back speech. The user operates the system through the terminal.

[1083] Overview of program processing

[1084] 1. Capture and transmit audio

[1085] Terminal

[1086] While the user is watching a television program, the terminal captures the television sound in real time using a built-in microphone or a connected external device.

[1087] The captured audio data is periodically sent to a server.

[1088] 2. Speech-to-text

[1089] server

[1090] The server receives the voice data sent from the device and sends it to the voice analysis API.

[1091] The voice analysis API analyzes voice data and converts it into text data.

[1092] The server receives the converted text data and formats it.

[1093] 3. Real-time subtitle generation and display

[1094] Terminal

[1095] The formatted text data is sent to the terminal and displayed in real time on the user interface.

[1096] 4. User's voice conversion request

[1097] User

[1098] The user selects a particular portion of the displayed text and requests to hear that text aloud.

[1099] Terminal

[1100] This request is sent to the server along with the text data.

[1101] 5. Text-to-speech

[1102] server

[1103] The server sends the text data selected by the user to the speech generation API.

[1104] The speech generation API analyzes text data and converts it into speech data.

[1105] The server receives the generated audio data.

[1106] 6. Playing audio data

[1107] Terminal

[1108] The audio data is sent to the terminal and played back via an audio player.

[1109] Specific examples

[1110] Example 1: For a hearing impaired user

[1111] Voice input

[1112] A user is watching a news program. During this time, the device captures the audio of the news and periodically transmits it to a server.

[1113] Text

[1114] The server sends the audio data of the news program to a speech analysis API and obtains text data such as, "Tonight's news will report on important events from around the world."

[1115] Real-time subtitle display

[1116] The captured text data is sent to the terminal and displayed on the user interface, allowing the user to read the subtitles displayed on the screen.

[1117] Example 2: For foreign users who do not understand Japanese

[1118] Voice input

[1119] A user is watching a variety show. The device captures the audio of the variety show and sends it to the server.

[1120] Text

[1121] The server sends the audio data from the variety show to the audio analysis API and obtains text data such as, "This week's guest is a world-famous artist."

[1122] Voice conversion request

[1123] The user selects the subtitle portion, "This week's guest is a world-class artist," to hear it in English.

[1124] Voice generation

[1125] The server sends the selected text data to the speech generation API and obtains English speech data.

[1126] Audio playback

[1127] The terminal plays back the acquired English audio data, allowing the user to understand the content.

[1128] The above is an example of a specific embodiment of the system according to the present invention, which aims to meet the diverse needs of users and make television viewing more accessible.

[1129] The processing flow will be explained below.

[1130] Step 1:

[1131] The device captures the audio of the TV program being watched by the user in real time using a built-in microphone or an external device, and stores the captured audio data in a buffer.

[1132] Step 2:

[1133] The terminal transmits the buffered audio data to the server at regular time intervals (for example, every second) using a protocol such as an HTTP POST request.

[1134] Step 3:

[1135] The server prepares to process the audio data received from the device and creates a request to send the audio data to the audio analysis API.

[1136] Step 4:

[1137] The server sends the voice data to the voice analysis API, which converts the voice data into text data.

[1138] Step 5:

[1139] The server receives the text data returned from the speech analysis API, formats the received text data, and adds timestamps, etc., as needed.

[1140] Step 6:

[1141] The server sends the formatted text data to the device using a WebSocket connection or an HTTP response.

[1142] Step 7:

[1143] The terminal acquires the text data received from the server and displays the acquired text data as real-time subtitles on the user interface.

[1144] Step 8:

[1145] The user selects a specific portion of the displayed subtitles and clicks on the corresponding interface button when he or she wishes to hear the text spoken.

[1146] Step 9:

[1147] The terminal receives the user's selection operation and transmits the selected text data and a voice conversion request to the server.

[1148] Step 10:

[1149] The server creates a request to send the selected text data received from the device to the speech generation API.

[1150] Step 11:

[1151] The server sends the text data to the speech generation API and requests speech conversion, which converts the text data into speech data.

[1152] Step 12:

[1153] The server receives the voice data returned from the voice generation API and sends it to the device as is.

[1154] Step 13:

[1155] The terminal acquires the audio data received from the server and plays the acquired audio data using an audio player so that the user can hear it.

[1156] The above are the specific steps of the system's processing, and the specific operations at each step are described, which allows a clear understanding of the system's operations.

[1157] Example 1

[1158] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1159] There is a need for a system that can convert television audio into text in real time and provide it to users in an easy-to-understand manner. In particular, users with hearing impairments and foreign users who speak different languages ​​need assistance in understanding television content. There is also a need for a system that allows users to select specific text and listen to it aloud.

[1160] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1161] In this invention, the server includes means for capturing audio data of a television broadcast, means for transmitting the captured audio data to the server, means for the server to transmit the audio data to an audio analysis API and format text data obtained from the audio analysis API, means for displaying the formatted text data on a user interface in real time, means for receiving a text selection from a user and transmitting the selected text data to an audio generation API, and means for playing the audio data obtained from the audio generation API, thereby enabling real-time conversion of audio to text and further playback of the user-selected text by audio.

[1162] "Means for capturing audio from a television broadcast" refers to means, including built-in microphones and external devices, used to collect audio from a television broadcast in real time.

[1163] The "means for transmitting captured audio data to a server" refers to a means having a function for periodically dividing the collected audio data into batches and transmitting the batches to a server.

[1164] The "means for the server to transmit voice data to the voice analysis API" refers to a means including a process of transmitting voice data received by the server to the voice analysis API and converting the voice data into text data.

[1165] "Means for formatting text data obtained from a speech analysis API" refers to means for formatting text data obtained from a speech analysis API and for separating sentences and breaking lines to make the data easier to read.

[1166] "Means for displaying formatted text data in a user interface in real time" refers to means that has the function of displaying formatted text data in a user interface in real time, and includes web pages, mobile applications, etc.

[1167] The "means for receiving a user's text selection and transmitting the selected text data to the speech generation API" refers to a means that, when a user selects a specific portion of displayed text, transmits the selected text data to the speech generation API.

[1168] "Means for playing back audio data obtained from an audio generation API" refers to means for playing back audio data obtained from an audio generation API on a terminal, and includes an audio player.

[1169] MODE FOR CARRYING OUT THE INVENTION

[1170] overview

[1171] This system converts television broadcast audio into text in real time and displays the text on a user interface. It also has the ability to convert user-selected text into speech and play it back. This system is implemented using a speech analysis API and a speech generation API.

[1172] Major hardware and software used

[1173] Hardware:

[1174] Device: Built-in microphone or external device (e.g. USB microphone)

[1175] software:

[1176] Speech analysis API: For example, Google Cloud Speech-to-Text API

[1177] Speech generation API: For example, Amazon Polly

[1178] User interface: HTML, JavaScript, and a front-end framework such as React or Vue.js

[1179] Example of operation

[1180] Audio capture and transmission

[1181] Terminal

[1182] While a user is watching a TV program, the device captures the TV audio in real time through a built-in microphone or an external device. Specifically, an audio capture application on the device monitors the microphone input and acquires audio data in real time.

[1183] The captured audio data is periodically split into small batches and sent to the server, for example every 5 seconds.

[1184] Speech to text

[1185] server

[1186] The server receives the voice data sent from the device via an HTTP request, and sends the received voice data to a voice analysis API (e.g., Google Cloud Speech-to-Text API).

[1187] The speech analysis API converts the speech data into text data, and the server receives the converted text data and performs any necessary formatting, such as adding sentence breaks and line breaks to make the text easier to read.

[1188] Real-time subtitle display

[1189] Terminal

[1190] The formatted text data is sent from the server to the terminal. Specifically, the server sends the text data as an HTTP response.

[1191] The device displays this text data in a user interface (e.g., web page, mobile app) in real time. Using JavaScript or a front-end framework, the text can be displayed dynamically.

[1192] User's voice conversion request

[1193] User

[1194] The user selects a particular portion of the displayed text, for example by clicking on a block of text.

[1195] Terminal

[1196] Based on this selection, the text data and a request for speech conversion are sent to the server. Specifically, the selected text and the request type (e.g., speech conversion request) are sent to the server as a POST request.

[1197] Text-to-speech

[1198] server

[1199] The server receives the text data selected by the user and extracts it from the JSON body of the HTTP request.

[1200] Send the received text data to a speech generation API (e.g., Amazon Polly), authenticating with an API key and specifying the required parameters (e.g., language, tone of voice).

[1201] The speech generation API analyzes the text data and converts it into speech data, and the server receives the generated speech data.

[1202] Playing audio data

[1203] Terminal

[1204] The generated voice data is sent from the server to the terminal, usually as an HTTP response.

[1205] The device receives this audio data and plays it using an audio player (e.g. HTML5 Audio). JavaScript is used to set the URL of the audio data in the audio player, and the audio is played when the play button is pressed.

[1206] Examples of prompt statements

[1207] "Please explain in natural language a system that converts audio from a news program into text and plays back the text selected by the user as speech."

[1208] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1209] Step 1: Capture and send audio

[1210] Terminal

[1211] While the user is watching a TV program, the device captures TV audio in real time using a built-in microphone or an external device. The captured audio data is divided into batches every 5 seconds. The device periodically sends these audio data batches to the server. The input is the captured audio data, and the output is the batched audio data. This capture process is performed using the Audio Capture API, and transmission to the server is via an HTTP POST request.

[1212] Step 2: Speech to text

[1213] server

[1214] The server receives the audio data batches sent from the device. It sends the received audio data batches to a speech analysis API and converts the audio data into text data. Here, the Google Cloud Speech-to-Text API is used. The converted text data is received and formatted. Specifically, it adds sentence separators and line breaks to make it easier to read. The input is the audio data batches, and the output is formatted text data.

[1215] Step 3: Display real-time subtitles

[1216] Terminal

[1217] The terminal receives formatted text data sent from the server. The terminal displays this text data in the user interface in real time. The text is displayed dynamically using JavaScript and a front-end framework (e.g., React, Vue.js). The input is formatted text data, and the output is real-time subtitles displayed in the user interface.

[1218] Step 4: User's voice conversion request

[1219] User

[1220] The user selects a specific portion of the displayed text, for example by clicking on the text block. The input is the text data displayed in real time, and the output is the text data selected by the user.

[1221] Step 5: Sending the selected text

[1222] Terminal

[1223] It receives the text data selected by the user and sends it to the server. Specifically, it sends the selected text data and the speech conversion request information to the server as a POST request. The input is the text data selected by the user, and the output is an HTTP request to the server.

[1224] Step 6: Convert text to speech

[1225] server

[1226] The server sends the received selected text data to the speech generation API. Amazon Polly is used here. Authentication is performed using an API key, and a speech conversion request is sent specifying the required parameters (e.g., language, tone of voice). The server receives the speech data obtained from the speech generation API. The input is the selected text data, and the output is the generated speech data.

[1227] Step 7: Playing back audio data

[1228] Terminal

[1229] Receives audio data sent from the server. The device receives the audio data and plays it using an audio player (e.g. HTML5 Audio). JavaScript is used to set the URL of the audio data in the audio player, and the audio is played when the play button is pressed. The input is the received audio data, and the output is the audio played to the user.

[1230] (Application example 1)

[1231] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1232] Conventional food delivery applications lack effective means of conveying audio information, such as restaurant promotional videos and menu introductions, to users. This poses a challenge, making it difficult to provide sufficient information to users with hearing impairments or who have difficulty understanding foreign languages. Furthermore, the lack of information provision through audiovisual interaction raises concerns about a decline in the quality of the user experience.

[1233] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1234] In this invention, the server includes means for transmitting voice data to a voice analysis API and acquiring text data, means for formatting the acquired text data and assigning a timestamp, means for displaying the text data on a user interface, means for receiving a user's text selection and transmitting it to a voice generation API to acquire voice data, means for playing the voice data, and means for capturing and converting the voice of an advertising medium into text to provide food delivery information, and converting the text selected by the user into voice and playing it back. This allows the user to visually check the voice information converted into text in real time, and further select the text as needed and play it back as voice information.

[1235] "Information media" refers to means of providing information through audio and video. This includes television broadcasts, radio broadcasts, online videos, etc.

[1236] A "voice analysis API" is an application programming interface (API) for analyzing voice data and converting it into text data. It is generally implemented using a cloud-based service.

[1237] A "user interface" is an interface that allows a user to interact with a system. It includes means such as a screen display and audio output.

[1238] A "Speech Generation API" is an application programming interface (API) for converting text data into speech data. It provides text-to-speech functionality.

[1239] "Audio data" means a digital representation of sound waves, typically captured by a microphone or recorded audio.

[1240] "Text data" refers to data written in characters. This includes character strings generated from audio data using a voice analysis API.

[1241] "Food delivery information" refers to information about food and drink delivery, including restaurant promotional videos, menu introductions, delivery status, etc.

[1242] "Promotional media" refers to the means of widely disseminating product information. This includes videos, audio, posters, etc.

[1243] This invention is a system that converts audio from television broadcasts and other information media into text in real time and displays it on a user interface. It also has the function of converting user-selected text into audio and playing it back. The following are specific examples of how this system can be implemented.

[1244] System Configuration

[1245] The system consists of three main components: a server, a terminal, and a user.

[1246] server

[1247] The server operates a program that uses the voice analysis API and voice generation API, which provides the following functions:

[1248] Send the audio data to the audio analysis API and obtain the text data.

[1249] The acquired text data is formatted and a timestamp is added.

[1250] The formatted text data is sent to the terminal and displayed.

[1251] It receives a text selection request from the user, sends the text to the speech generation API, and obtains the speech data.

[1252] Terminal

[1253] A terminal is a device (e.g., a smartphone or tablet) that is operated by a user. A terminal performs the following functions:

[1254] Capture audio using the built-in microphone or an external device.

[1255] Send the captured audio data to the server.

[1256] The text data transmitted from the server is displayed on the user interface in real time.

[1257] The text data selected by the user is sent again to the server, and the voice data is acquired.

[1258] The acquired audio data is played back.

[1259] User

[1260] The User is the final user of the system and performs the following operations:

[1261] Watch television broadcasts and other information media and capture audio through your device.

[1262] View real-time text data displayed in the user interface.

[1263] Select a specific portion of text and make a speech conversion request.

[1264] Listen to the audio being played and understand the information.

[1265] Program processing procedure

[1266] The server first receives the voice data sent from the device. Next, it sends the voice data to a voice analysis API to obtain text data. This text data is formatted and a timestamp is added. The formatted text is sent to the device and displayed in the user interface. When the user selects a portion of the text and requests voice conversion, the server sends the text data to a voice generation API to obtain the voice data. Finally, the voice data is sent to the device and played.

[1267] Hardware and software used

[1268] Hardware

[1269] Smartphones, tablets, and microphones

[1270] software

[1271] Speech analysis API (e.g., Google Cloud Speech-to-Text)

[1272] Speech generation APIs (e.g., Google Text-to-Speech)

[1273] Server-side program (implemented in Python etc.)

[1274] User Interface (using a front-end framework)

[1275] Specific examples

[1276] When a user uses a food delivery app, the audio from a restaurant's promotional video is captured by the smartphone's microphone. This audio is converted into text in real time and displayed on the user interface. The user selects the part of the displayed text that says "Today's special is sushi pizza" and requests that it be played aloud. The server sends this text to a speech generation API, which then transmits the generated audio to the device. Finally, the user hears the audio saying "sushi pizza."

[1277] Prompt Sentence Examples

[1278] "When a user types "sushi pizza" into your application, convert the text into speech and play it back."

[1279] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1280] Output results according to processing steps

[1281] Step 1:

[1282] The device captures audio from information media using a built-in microphone or an external device, and this audio data is collected in real time and sent to a server for further processing.

[1283] Input: Audio data (audio from information media)

[1284] Output: Audio data is sent to the server

[1285] Step 2:

[1286] The server receives the voice data sent from the device, then uses a voice analysis API to convert the voice data into text data, which is then formatted and timestamped.

[1287] Input: Audio data

[1288] Output: Formatted text data (with timestamp)

[1289] Step 3:

[1290] The server sends the formatted text data to the terminal.

[1291] Input: Formatted text data

[1292] Output: Text data is sent to the terminal

[1293] Step 4:

[1294] The terminal displays the transmitted text data on the user interface in real time, allowing the user to check the text data displayed in a visible format.

[1295] Input: Text data

[1296] Output: Real-time displayed text data

[1297] Step 5:

[1298] The user selects the desired part from the text data, and the selected text is sent back to the server as a speech generation request.

[1299] Input: Real-time displayed text data

[1300] Output: The selected text data is sent to the server.

[1301] Step 6:

[1302] The server uses a speech generation API to convert the selected text data into speech data, which is then sent to the device.

[1303] Input: Selected text data

[1304] Output: Generated audio data

[1305] Step 7:

[1306] The terminal plays the audio data that has been sent, and the user can hear the audio corresponding to the selected text.

[1307] Input: Generated audio data

[1308] Output: Played audio data (user hears the audio)

[1309] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1310] Summary of the Invention

[1311] This invention is a system that converts television broadcast audio into text in real time and displays that text on a user interface. It also has the ability to convert user-selected text into speech and play it back. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, the user experience can be further improved. This system is realized using a speech analysis API and a speech generation API.

[1312] System Configuration

[1313] This system consists of three main components: a server, a terminal, and a user. The server is responsible for converting speech to text and generating speech, while the terminal is responsible for capturing speech, displaying text, playing speech, and analyzing emotions using an emotion engine. The user operates the system through the terminal.

[1314] Overview of program processing

[1315] 1. Capture and transmit audio

[1316] Terminal

[1317] While the user is watching a television program, the terminal captures the television sound in real time using a built-in microphone or a connected external device.

[1318] The captured audio data is periodically sent to a server.

[1319] 2. Speech-to-text

[1320] server

[1321] The server receives the voice data sent from the device and sends it to the voice analysis API.

[1322] The voice analysis API analyzes voice data and converts it into text data.

[1323] The server receives the converted text data and formats it.

[1324] 3. Real-time subtitle generation and display

[1325] Terminal

[1326] The formatted text data is sent to the terminal and displayed in real time on the user interface.

[1327] 4. User's voice conversion request

[1328] User

[1329] The user selects a particular portion of the displayed text and requests to hear that text aloud.

[1330] Terminal

[1331] This request is sent to the server along with the text data.

[1332] 5. Text-to-speech

[1333] server

[1334] The server sends the text data selected by the user to the speech generation API.

[1335] The speech generation API analyzes text data and converts it into speech data.

[1336] The server receives the generated audio data.

[1337] 6. Playing audio data

[1338] Terminal

[1339] The audio data is sent to the terminal and played back via an audio player.

[1340] 7. Emotion Analysis Using an Emotion Engine

[1341] Terminal

[1342] The device captures the user's facial expressions or tone of voice in real time and sends them to the emotion engine.

[1343] The emotion engine analyzes the user's emotions and returns the results to the terminal.

[1344] 8. Emotion-Based Interface Adjustment

[1345] Terminal

[1346] Based on the analysis results from the emotion engine, the content of the text data displayed and the tone and tempo of the audio data are adjusted.

[1347] Specific examples

[1348] Example 1: For a hearing impaired user

[1349] Voice input

[1350] A user is watching a news program. During this time, the device captures the audio of the news and periodically transmits it to a server.

[1351] Text

[1352] The server sends the audio data of the news program to a speech analysis API and obtains text data such as, "Tonight's news will report on important events from around the world."

[1353] Real-time subtitle display

[1354] The captured text data is sent to the terminal and displayed on the user interface, allowing the user to read the subtitles displayed on the screen.

[1355] Sentiment analysis and interface adjustment

[1356] The device sends the user's facial expressions and tone of voice to an emotion engine, which then adjusts the display content and the tone and tempo of the audio playback based on the analysis results.

[1357] Example 2: For foreign users who do not understand Japanese

[1358] Voice input

[1359] A user is watching a variety show. The device captures the audio of the variety show and sends it to the server.

[1360] Text

[1361] The server sends the audio data from the variety show to the audio analysis API and obtains text data such as, "This week's guest is a world-famous artist."

[1362] Voice conversion request

[1363] The user selects the subtitle portion, "This week's guest is a world-class artist," to hear it in English.

[1364] Voice generation

[1365] The server sends the selected text data to the speech generation API and obtains English speech data.

[1366] Audio playback and emotion analysis

[1367] The device plays back the acquired English audio data, while simultaneously analyzing the user's emotions using an emotion engine to adjust the tone and tempo of the audio playback.

[1368] The above is an example of a specific embodiment of the system according to the present invention, which aims to meet the diverse needs of users and make television viewing more accessible and personalized.

[1369] The processing flow will be explained below.

[1370] Step 1:

[1371] The device captures the audio of the TV program being watched by the user in real time using a built-in microphone or an external device, and stores the captured audio data in a buffer.

[1372] Step 2:

[1373] The terminal transmits the buffered audio data to the server at regular time intervals (for example, every second) using a protocol such as an HTTP POST request.

[1374] Step 3:

[1375] The server prepares to process the audio data received from the device and creates a request to send the audio data to the audio analysis API.

[1376] Step 4:

[1377] The server sends the voice data to the voice analysis API, which converts the voice data into text data.

[1378] Step 5:

[1379] The server receives the text data returned from the speech analysis API, formats the received text data, and adds timestamps, etc., as needed.

[1380] Step 6:

[1381] The server sends the formatted text data to the device using a WebSocket connection or an HTTP response.

[1382] Step 7:

[1383] The terminal acquires the text data received from the server and displays the acquired text data as real-time subtitles on the user interface.

[1384] Step 8:

[1385] The device captures the user's facial expressions or tone of voice in real time and sends them to the emotion engine, which analyzes the user's emotions and sends the results back to the device.

[1386] Step 9:

[1387] The device adjusts the displayed content of the text data and the tone and tempo of the voice based on the analysis results from the emotion engine. If the user's emotion is positive, the device brightens the displayed text and brightens the tone of the voice.

[1388] Step 10:

[1389] The user selects a specific portion of the displayed subtitles and clicks on the corresponding interface button when he or she wishes to hear the text spoken.

[1390] Step 11:

[1391] The terminal receives the user's selection operation and transmits the selected text data and a voice conversion request to the server.

[1392] Step 12:

[1393] The server creates a request to send the selected text data received from the device to the speech generation API.

[1394] Step 13:

[1395] The server sends the text data to the speech generation API and requests speech conversion, which converts the text data into speech data.

[1396] Step 14:

[1397] The server receives the voice data returned from the voice generation API and sends it to the device as is.

[1398] Step 15:

[1399] The terminal acquires the audio data received from the server and plays the acquired audio data using an audio player so that the user can hear it.

[1400] The above are the specific steps of the system's processing, and the specific operations at each step are described, allowing the system's operations to be clearly understood.

[1401] Example 2

[1402] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1403] In conventional television broadcast viewing systems, there are users who cannot properly receive audio information due to hearing impairments or language barriers. For this reason, there is a need for a method to convert audio into text in real time and display that text to the user. However, existing systems have difficulty providing a personalized interface that takes user emotions into account. In addition, there is also the challenge of seamlessly capturing audio data in real time, converting it into text, and playing back the text as audio.

[1404] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a voice analysis means, a user interface means, a voice generation means, an emotion analysis means, and an interface adjustment means. This makes it possible to convert voice into text in real time and provide the text to the user in a personalized format. In addition, the process from capturing voice data to converting it into text and then playing the voice can be seamlessly linked, improving the user experience.

[1405] The "voice analysis means" is a means for analyzing voice data and converting it into text data.

[1406] The "user interface means" is a means for displaying text data to a user in real time and providing an interface for the user to input data.

[1407] The "voice generating means" is a means for analyzing text data and converting it into voice data.

[1408] The "emotion analysis means" is a means for capturing the user's facial expressions and tone of voice, and analyzing the data to recognize emotions.

[1409] The "interface adjustment means" is a means for adjusting the user interface, tone, tempo, etc. of audio playback based on the analyzed emotions.

[1410] System Overview

[1411] The present invention is a system that converts television broadcast audio into text in real time and displays the text on a user interface. It also has the function of converting user-selected text into speech and playing it back. The system can improve the user experience by recognizing the user's emotions and adjusting the interface based on them. This system is realized using a speech analysis means and a speech generation means.

[1412] Hardware and software used

[1413] Hardware:

[1414] Client device (terminal): Captures TV audio and displays text and plays audio.

[1415] Server: Analyzes and generates audio data.

[1416] software:

[1417] Speech analysis API: Analyzes voice data and converts it into text data (for example, Google Cloud Speech-to-Text API).

[1418] Speech generation API: Analyzes text data and converts it into speech data (e.g., Amazon Polly).

[1419] Emotion analysis API: Analyzes a user's facial expressions and tone of voice to recognize their emotions (for example, Microsoft Azure Cognitive Services Emotion API).

[1420] Program processing

[1421] 1. Capture and transmit audio

[1422] Terminal

[1423] While a user is watching a TV program, the device captures the TV audio in real time using a built-in microphone or a connected external device, and the captured audio data is periodically sent to a server.

[1424] 2. Speech-to-text

[1425] server

[1426] The server receives the voice data sent from the device and sends it to the voice analysis API. The voice analysis API analyzes the voice data and converts it into text data. The server then formats the converted text data and makes any necessary corrections to fit the context.

[1427] 3. Real-time subtitle generation and display

[1428] Terminal

[1429] Receives formatted text data sent from the server and displays it in the user interface in real time. Adjusts the font size and color of the text so that the user can easily read the text data.

[1430] 4. User's voice conversion request

[1431] User

[1432] The user selects a particular portion of text displayed on the screen and requests to hear the selected text aloud.

[1433] Terminal

[1434] It receives a user request and sends the text data to the server.

[1435] 5. Text-to-speech

[1436] server

[1437] The server sends the text data selected by the user to the speech generation API. The speech generation API analyzes the text data and converts it into speech data. The server receives the generated speech data and sends it back to the device.

[1438] 6. Playing audio data

[1439] Terminal

[1440] The device decompresses the received audio data and plays it through an audio player, providing an interface that allows you to adjust the volume and speed of the audio being played.

[1441] 7. Emotion Analysis Using an Emotion Engine

[1442] Terminal

[1443] The device captures the user's facial expressions or tone of voice using a built-in camera or microphone, and sends the captured data to the emotion engine, which analyzes the user's emotions in real time and sends the results back to the device.

[1444] 8. Emotion-Based Interface Adjustment

[1445] Terminal

[1446] Based on the analysis results from the emotion engine, the content displayed in the user interface and the tone and tempo of the audio data can be adjusted. For example, if the user shows a confused expression, the text font can be made larger and the audio playback speed can be slowed down.

[1447] Specific examples and prompts for the generative AI model

[1448] Example 1: A hearing-impaired user

[1449] Voice input

[1450] A user is watching a news program. During this time, the device captures the audio of the news and periodically transmits it to a server.

[1451] Text

[1452] The server sends the audio data of the news program to the audio analysis API and obtains the text data, "Tonight's news will report on important events from around the world."

[1453] Real-time subtitle display

[1454] The captured text data is sent to the terminal and displayed on the user interface, allowing the user to read the subtitles displayed on the screen.

[1455] Sentiment analysis and interface adjustment

[1456] The device sends the user's facial expressions and tone of voice to an emotion engine, which then adjusts the display content and the tone and tempo of the audio playback based on the analysis results.

[1457] Example 2: For foreign users who do not understand Japanese

[1458] Voice input

[1459] A user is watching a variety show. The device captures the audio of the variety show and sends it to the server.

[1460] Text

[1461] The server sends the audio data from the variety show to the audio analysis API and obtains text data such as "This week's guest is a world-famous artist."

[1462] Voice conversion request

[1463] The user selects the subtitle portion, "This week's guest is a world-class artist," to hear it in English.

[1464] Voice generation

[1465] The server sends the selected text data to the speech generation API and obtains English speech data.

[1466] Audio playback and emotion analysis

[1467] The device plays back the acquired English audio data, while simultaneously analyzing the user's emotions using an emotion engine to adjust the tone and tempo of the audio playback.

[1468] Example prompts for generative AI models

[1469] 1. Example prompts for transcription of news programs

[1470] "Capture the audio of news programs and convert it into text data in real time using speech analysis methods and display it."

[1471] 2. Example prompts for sentiment analysis

[1472] "Capture the user's facial expressions and tone of voice, use sentiment analysis tools to perform sentiment analysis, and adjust the interface based on the results."

[1473] The above is an example of a specific embodiment of the system according to the present invention, which aims to meet the diverse needs of users and make television viewing more accessible and personalized.

[1474] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1475] Step 1:

[1476] Audio capture and transmission

[1477] Terminal

[1478] While the user is watching a TV program, the device captures the TV audio in real time using the built-in microphone or a connected external device. This inputs the TV audio into the device as audio data. The captured audio data is temporarily stored in buffer memory. The audio data is then compressed at regular intervals (e.g., every 5 seconds) and sent to the server.

[1479] Step 2:

[1480] Speech to text

[1481] server

[1482] The server receives the voice data sent from the terminal. It then sends the received voice data to a voice analysis means (e.g., a voice analysis API). The voice analysis means analyzes the voice data and converts it into text data. During this process, the voice signal is digitized and context estimation is performed in conjunction with phonemic analysis. After the converted text data is sent back to the server, the server formats the text data and corrects it into an easy-to-read format. This formatted text data is then output.

[1483] Step 3:

[1484] Real-time subtitle generation and display

[1485] Terminal

[1486] The formatted text data sent from the server is received. The received text data is displayed in real time on the user interface. When displayed, the size and color of the characters, the transparency of the background, etc. are adjusted to make it easy for the user to read. At this time, the text data is input into the user interface and output in a visually represented form.

[1487] Step 4:

[1488] User's voice conversion request

[1489] User

[1490] The user selects a specific portion of text displayed on the screen. For example, they select a segment of a news program: "Tonight's news will report on important events from around the world." This selection is entered into the device's interface. The user then makes a request to hear the selected text aloud. This request triggers the transmission of the selected text data from the device to the server.

[1491] Step 5:

[1492] Text-to-speech

[1493] server

[1494] The server receives the text data selected by the user. This text data is sent to a speech generation means (e.g., a speech generation API). The speech generation means analyzes the text data and converts it into speech data. Specifically, it performs text analysis and generates a speech file using a speech synthesis engine. This generated speech data is sent back to the server, and then retransmitted from the server to the terminal.

[1495] Step 6:

[1496] Playing audio data

[1497] Terminal

[1498] The device decompresses the audio data received from the server and plays it through the built-in audio player. This audio data is played in an audio format (e.g., MP3) and output to the user's ears. The user has an interface to adjust the volume and speed of the audio being played.

[1499] Step 7:

[1500] Emotion analysis using an emotion engine

[1501] Terminal

[1502] The device uses a built-in camera and microphone to capture the user's facial expressions and tone of voice. This captured data is sent to an emotion analysis means (e.g., emotion analysis API). The emotion analysis means analyzes this data and identifies the user's emotions (e.g., joy, sadness, anger, confusion, etc.). The analysis results are sent back to the device in real time and output as the user's emotion data.

[1503] Step 8:

[1504] Emotion-based interface adjustment

[1505] Terminal

[1506] Based on the analysis results received from the emotion analysis means, the device adjusts the content displayed in the user interface and the tone and tempo of the audio data. For example, if the user shows a confused expression, the device increases the font size of the text and slows down the audio playback speed. This series of adjustment operations is reflected in the user interface and is output as an optimized interface.

[1507] This is the specific flow of the program processing for this system. The inputs and outputs at each processing step, as well as the specific operations, are clearly shown, allowing for a more detailed understanding of the operation of the entire system.

[1508] (Application example 2)

[1509] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1510] By converting television broadcast audio into text in real time and displaying it on a user interface, users with hearing impairments or difficulty understanding foreign languages ​​can easily understand the broadcast content. However, this system faces several challenges. For example, if a user selects a specific text section and wants to hear it aloud, fast and accurate speech conversion is required. Furthermore, to more individually optimize the user experience, it is necessary to analyze the user's emotions and state in real time and adjust the interface and playback speed accordingly.

[1511] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for capturing audio from a television broadcast, means for transmitting the captured audio data to a voice analysis API, means for displaying text data acquired from the voice analysis API on a user interface in real time, means for receiving a user's text selection and transmitting the selected text data to a voice generation API, means for playing back the audio data acquired from the voice generation API, means for capturing the user's facial expression or tone of voice and transmitting it to an emotion analysis engine, and means for adjusting the interface and playback speed based on the analysis results of the emotion analysis engine. This not only meets the diverse needs of users, but also makes it possible to optimize the user experience in a more personalized manner.

[1512] "Television broadcasting" is a broadcasting format that transmits images and sounds to ordinary households or specific regions using radio waves or cables.

[1513] "Audio capture" is the process of collecting audio data in digital form using a microphone or other audio input device.

[1514] "Audio data" is a digital representation of captured audio in a data format that can be analyzed and processed.

[1515] A "voice analysis API" is a programming interface for analyzing voice data and converting it into text data.

[1516] "Text data" is information stored in the form of character string data, and is used for displaying on a user interface or converting into voice.

[1517] "User interface" is a general term for the screens and input / output devices that allow users to interact with a system.

[1518] A "voice generation API" is a programming interface for converting text data into voice data.

[1519] "Audio data playback" is the process of audibly outputting digital audio data through an audio playback device.

[1520] An "emotion analysis engine" is a software component that analyzes a user's facial expressions and tone of voice to recognize their emotional state.

[1521] "Interface adjustment" is the process of changing the parameters of the user interface and audio playback based on the analysis results of the emotion analysis engine.

[1522] "Time-stamped text" refers to text data to which specific time information has been added, and corresponds to a specific portion of audio content.

[1523] MODE FOR CARRYING OUT THE INVENTION

[1524] Summary of the Invention Embodiments

[1525] The present invention relates to a system that converts television broadcast audio into text in real time and displays the text on a user interface. This system analyzes audio data captured during television viewing and generates text data. Furthermore, the system has a function that allows a user to select a specific text portion, converts the selected text into audio, and plays it back. Furthermore, by combining it with an emotion analysis engine, the system can adjust the interface and audio playback based on the user's emotions.

[1526] System Configuration

[1527] This system consists of three main components: a server, a terminal, and a user.

[1528] server:

[1529] Use a speech analysis API to convert audio data into text.

[1530] Uses the speech generation API to convert the selected text into audio data.

[1531] Device:

[1532] It captures audio data from television broadcasts in real time and displays the text.

[1533] It receives the user's voice requests and sends them to the server.

[1534] It uses a sentiment analysis engine to analyze user emotions and adjust the interface and audio playback accordingly.

[1535] User:

[1536] Watch a TV broadcast and select the displayed text to request a voice conversion.

[1537] Emotions are input through facial expressions and tone of voice.

[1538] Program processing

[1539] Audio capture and transmission

[1540] The device captures audio from a television broadcast using a built-in or external microphone and periodically transmits the captured audio to a server. For example, while watching a news program, the device continuously captures the audio of the news.

[1541] Speech to text

[1542] The server sends the captured audio data to a speech analysis API (such as the Google Speech API) and converts it into text data. This process generates text data such as "Tonight's news will report on important events from around the world."

[1543] Real-time text display

[1544] The generated text data is sent from the server to the terminal and displayed in real time on the user interface, allowing the user to visually confirm the news anchor's remarks as text in real time.

[1545] User's voice conversion request

[1546] The user selects a specific text segment and sends a request to the server via the terminal to listen to it aloud. For example, if the user selects "This week's guest is a world-famous artist," a voice conversion request for that segment is sent.

[1547] Text-to-speech

[1548] The server sends the selected text data to a speech generation API (e.g., Amazon Polly) to generate corresponding voice data, which is then sent back to the device.

[1549] Playing audio data

[1550] The device plays the received audio data through an audio player. In this case, after selecting the text "This week's guest is a world-famous artist," the user can listen to the content in English.

[1551] Analysis by sentiment analysis engine

[1552] The device captures the user's facial expressions and tone of voice and sends them to an emotion analysis engine (using OpenCV and FER, for example). Based on the analysis results, the device adjusts the user interface, audio playback speed, etc.

[1553] Prompt Sentence Examples

[1554] "Demonstrate an application that translates captured text in Japanese in real time and adjusts the interface and audio playback based on the user's emotions."

[1555] This makes it possible to provide a system that meets the diverse needs of users and makes television viewing more accessible and personalized.

[1556] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1557] Program processing flow

[1558] Step 1:

[1559] Audio capture and transmission

[1560] The device uses a built-in or external microphone to capture the audio of the TV broadcast in real time. It then periodically transmits the captured audio data to a server. The input is the audio of the TV broadcast, and the output is digital audio data. Specifically, the device's microphone is activated, the audio data is temporarily stored in a buffer, and the buffer is periodically uploaded to the server.

[1561] Step 2:

[1562] Speech to text

[1563] The server sends the received voice data to the voice analysis API. The voice analysis API converts the voice data into text data and sends the text data back to the server. The input is voice data and the output is text data. Specifically, the server calls the voice analysis API, sends the voice data, and receives the text data returned from the API.

[1564] Step 3:

[1565] Real-time text display

[1566] Text data sent from the server is sent to the terminal and displayed in real time on the user interface. The input is text data from the server, and the output is the text displayed on the user interface. Specifically, the terminal receives the text data and immediately reflects it in the display area of ​​the user interface.

[1567] Step 4:

[1568] User's voice conversion request

[1569] The user selects a specific portion of the displayed text and requests to hear the selected text aloud. The input is the selected text on the user interface, and the output is the requested data. Specific actions require the user to select the text using a touch screen or mouse and press a request button.

[1570] Step 5:

[1571] Text-to-speech

[1572] Text data requested from the device is sent to the server, which then uses a speech generation API to convert the text data into voice data. The voice data returned from the voice generation API is stored on the server and sent to the device. The input is the selected text data, and the output is voice data. Specifically, the server calls the voice generation API, sends the text data, and receives the generated voice data.

[1573] Step 6:

[1574] Playing audio data

[1575] The device plays the received audio data through an audio player. The input is the audio data from the server, and the output is the played audio. Specifically, the device starts the audio player, decodes the received audio data, and plays it.

[1576] Step 7:

[1577] Analysis by sentiment analysis engine

[1578] The device captures the user's facial expressions and tone of voice in real time and sends them to an emotion analysis engine. The emotion analysis engine analyzes the user's emotions and sends the results back to the device. The input is the captured video or audio data, and the output is the analyzed emotion data. Specific operations include activating the device's camera, capturing the user's facial expressions, and sending the video data to the emotion analysis engine.

[1579] Step 8:

[1580] Emotion-based interface adjustment

[1581] Based on the analysis results from the emotion analysis engine, the device adjusts the tone and tempo of the user interface and audio playback. The input is emotion analysis data, and the output is adjusted interface elements and audio playback. Specifically, the device automatically changes the color scheme of the user interface and adjusts the playback speed according to the analysis results.

[1582] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1583] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1584] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1585] [Fourth embodiment]

[1586] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1587] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1588] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1589] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1590] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1591] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1592] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1593] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1594] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1595] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1596] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1597] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1598] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1599] Summary of the Invention

[1600] This system converts television broadcast audio into text in real time and displays the text on a user interface. It also has the ability to convert user-selected text into speech and play it back. This system is implemented using a speech analysis API and a speech generation API.

[1601] System Configuration

[1602] This system consists of three main components: a server, a terminal, and a user. The server is responsible for converting speech to text and generating speech, while the terminal captures speech, displays text, and plays back speech. The user operates the system through the terminal.

[1603] Overview of program processing

[1604] 1. Capture and transmit audio

[1605] Terminal

[1606] While the user is watching a television program, the terminal captures the television sound in real time using a built-in microphone or a connected external device.

[1607] The captured audio data is periodically sent to a server.

[1608] 2. Speech-to-text

[1609] server

[1610] The server receives the voice data sent from the device and sends it to the voice analysis API.

[1611] The voice analysis API analyzes voice data and converts it into text data.

[1612] The server receives the converted text data and formats it.

[1613] 3. Real-time subtitle generation and display

[1614] Terminal

[1615] The formatted text data is sent to the terminal and displayed in real time on the user interface.

[1616] 4. User's voice conversion request

[1617] User

[1618] The user selects a particular portion of the displayed text and requests to hear that text aloud.

[1619] Terminal

[1620] This request is sent to the server along with the text data.

[1621] 5. Text-to-speech

[1622] server

[1623] The server sends the text data selected by the user to the speech generation API.

[1624] The speech generation API analyzes text data and converts it into speech data.

[1625] The server receives the generated audio data.

[1626] 6. Playing audio data

[1627] Terminal

[1628] The audio data is sent to the terminal and played back via an audio player.

[1629] Specific examples

[1630] Example 1: For a hearing impaired user

[1631] Voice input

[1632] A user is watching a news program. During this time, the device captures the audio of the news and periodically transmits it to a server.

[1633] Text

[1634] The server sends the audio data of the news program to a speech analysis API and obtains text data such as, "Tonight's news will report on important events from around the world."

[1635] Real-time subtitle display

[1636] The captured text data is sent to the terminal and displayed on the user interface, allowing the user to read the subtitles displayed on the screen.

[1637] Example 2: For foreign users who do not understand Japanese

[1638] Voice input

[1639] A user is watching a variety show. The device captures the audio of the variety show and sends it to the server.

[1640] Text

[1641] The server sends the audio data from the variety show to the audio analysis API and obtains text data such as, "This week's guest is a world-famous artist."

[1642] Voice conversion request

[1643] The user selects the subtitle portion, "This week's guest is a world-class artist," to hear it in English.

[1644] Voice generation

[1645] The server sends the selected text data to the speech generation API and obtains English speech data.

[1646] Audio playback

[1647] The terminal plays back the acquired English audio data, allowing the user to understand the content.

[1648] The above is an example of a specific embodiment of the system according to the present invention, which aims to meet the diverse needs of users and make television viewing more accessible.

[1649] The processing flow will be explained below.

[1650] Step 1:

[1651] The device captures the audio of the TV program being watched by the user in real time using a built-in microphone or an external device, and stores the captured audio data in a buffer.

[1652] Step 2:

[1653] The terminal transmits the buffered audio data to the server at regular time intervals (for example, every second) using a protocol such as an HTTP POST request.

[1654] Step 3:

[1655] The server prepares to process the audio data received from the device and creates a request to send the audio data to the audio analysis API.

[1656] Step 4:

[1657] The server sends the voice data to the voice analysis API, which converts the voice data into text data.

[1658] Step 5:

[1659] The server receives the text data returned from the speech analysis API, formats the received text data, and adds timestamps, etc., as needed.

[1660] Step 6:

[1661] The server sends the formatted text data to the device using a WebSocket connection or an HTTP response.

[1662] Step 7:

[1663] The terminal acquires the text data received from the server and displays the acquired text data as real-time subtitles on the user interface.

[1664] Step 8:

[1665] The user selects a specific portion of the displayed subtitles and clicks on the corresponding interface button when he or she wishes to hear the text spoken.

[1666] Step 9:

[1667] The terminal receives the user's selection operation and transmits the selected text data and a voice conversion request to the server.

[1668] Step 10:

[1669] The server creates a request to send the selected text data received from the device to the speech generation API.

[1670] Step 11:

[1671] The server sends the text data to the speech generation API and requests speech conversion, which converts the text data into speech data.

[1672] Step 12:

[1673] The server receives the voice data returned from the voice generation API and sends it to the device as is.

[1674] Step 13:

[1675] The terminal acquires the audio data received from the server and plays the acquired audio data using an audio player so that the user can hear it.

[1676] The above are the specific steps of the system's processing, and the specific operations at each step are described, which allows a clear understanding of the system's operations.

[1677] Example 1

[1678] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1679] There is a need for a system that can convert television audio into text in real time and provide it to users in an easy-to-understand manner. In particular, users with hearing impairments and foreign users who speak different languages ​​need assistance in understanding television content. There is also a need for a system that allows users to select specific text and listen to it aloud.

[1680] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1681] In this invention, the server includes means for capturing audio data of a television broadcast, means for transmitting the captured audio data to the server, means for the server to transmit the audio data to an audio analysis API and format text data obtained from the audio analysis API, means for displaying the formatted text data on a user interface in real time, means for receiving a text selection from a user and transmitting the selected text data to an audio generation API, and means for playing the audio data obtained from the audio generation API, thereby enabling real-time conversion of audio to text and further playback of the user-selected text by audio.

[1682] "Means for capturing audio from a television broadcast" refers to means, including built-in microphones and external devices, used to collect audio from a television broadcast in real time.

[1683] The "means for transmitting captured audio data to a server" refers to a means having a function for periodically dividing the collected audio data into batches and transmitting the batches to a server.

[1684] The "means for the server to transmit voice data to the voice analysis API" refers to a means including a process of transmitting voice data received by the server to the voice analysis API and converting the voice data into text data.

[1685] "Means for formatting text data obtained from a speech analysis API" refers to means for formatting text data obtained from a speech analysis API and for separating sentences and breaking lines to make the data easier to read.

[1686] "Means for displaying formatted text data in a user interface in real time" refers to means that has the function of displaying formatted text data in a user interface in real time, and includes web pages, mobile applications, etc.

[1687] The "means for receiving a user's text selection and transmitting the selected text data to the speech generation API" refers to a means that, when a user selects a specific portion of displayed text, transmits the selected text data to the speech generation API.

[1688] "Means for playing back audio data obtained from an audio generation API" refers to means for playing back audio data obtained from an audio generation API on a terminal, and includes an audio player.

[1689] MODE FOR CARRYING OUT THE INVENTION

[1690] overview

[1691] This system converts television broadcast audio into text in real time and displays the text on a user interface. It also has the ability to convert user-selected text into speech and play it back. This system is implemented using a speech analysis API and a speech generation API.

[1692] Major hardware and software used

[1693] Hardware:

[1694] Device: Built-in microphone or external device (e.g. USB microphone)

[1695] software:

[1696] Speech analysis API: For example, Google Cloud Speech-to-Text API

[1697] Speech generation API: For example, Amazon Polly

[1698] User interface: HTML, JavaScript, and a front-end framework such as React or Vue.js

[1699] Example of operation

[1700] Audio capture and transmission

[1701] Terminal

[1702] While a user is watching a TV program, the device captures the TV audio in real time through a built-in microphone or an external device. Specifically, an audio capture application on the device monitors the microphone input and acquires audio data in real time.

[1703] The captured audio data is periodically split into small batches and sent to the server, for example every 5 seconds.

[1704] Speech to text

[1705] server

[1706] The server receives the voice data sent from the device via an HTTP request, and sends the received voice data to a voice analysis API (e.g., Google Cloud Speech-to-Text API).

[1707] The speech analysis API converts the speech data into text data, and the server receives the converted text data and performs any necessary formatting, such as adding sentence breaks and line breaks to make the text easier to read.

[1708] Real-time subtitle display

[1709] Terminal

[1710] The formatted text data is sent from the server to the terminal. Specifically, the server sends the text data as an HTTP response.

[1711] The device displays this text data in a user interface (e.g., web page, mobile app) in real time. Using JavaScript or a front-end framework, the text can be displayed dynamically.

[1712] User's voice conversion request

[1713] User

[1714] The user selects a particular portion of the displayed text, for example by clicking on a block of text.

[1715] Terminal

[1716] Based on this selection, the text data and a request for speech conversion are sent to the server. Specifically, the selected text and the request type (e.g., speech conversion request) are sent to the server as a POST request.

[1717] Text-to-speech

[1718] server

[1719] The server receives the text data selected by the user and extracts it from the JSON body of the HTTP request.

[1720] Send the received text data to a speech generation API (e.g., Amazon Polly), authenticating with an API key and specifying the required parameters (e.g., language, tone of voice).

[1721] The speech generation API analyzes the text data and converts it into speech data, and the server receives the generated speech data.

[1722] Playing audio data

[1723] Terminal

[1724] The generated voice data is sent from the server to the terminal, usually as an HTTP response.

[1725] The device receives this audio data and plays it using an audio player (e.g. HTML5 Audio). JavaScript is used to set the URL of the audio data in the audio player, and the audio is played when the play button is pressed.

[1726] Examples of prompt statements

[1727] "Please explain in natural language a system that converts audio from a news program into text and plays back the text selected by the user as speech."

[1728] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1729] Step 1: Capture and send audio

[1730] Terminal

[1731] While the user is watching a TV program, the device captures TV audio in real time using a built-in microphone or an external device. The captured audio data is divided into batches every 5 seconds. The device periodically sends these audio data batches to the server. The input is the captured audio data, and the output is the batched audio data. This capture process is performed using the Audio Capture API, and transmission to the server is via an HTTP POST request.

[1732] Step 2: Speech to text

[1733] server

[1734] The server receives the audio data batches sent from the device. It sends the received audio data batches to a speech analysis API and converts the audio data into text data. Here, the Google Cloud Speech-to-Text API is used. The converted text data is received and formatted. Specifically, it adds sentence separators and line breaks to make it easier to read. The input is the audio data batches, and the output is formatted text data.

[1735] Step 3: Display real-time subtitles

[1736] Terminal

[1737] The terminal receives formatted text data sent from the server. The terminal displays this text data in the user interface in real time. The text is displayed dynamically using JavaScript and a front-end framework (e.g., React, Vue.js). The input is formatted text data, and the output is real-time subtitles displayed in the user interface.

[1738] Step 4: User's voice conversion request

[1739] User

[1740] The user selects a specific portion of the displayed text, for example by clicking on the text block. The input is the text data displayed in real time, and the output is the text data selected by the user.

[1741] Step 5: Sending the selected text

[1742] Terminal

[1743] It receives the text data selected by the user and sends it to the server. Specifically, it sends the selected text data and the speech conversion request information to the server as a POST request. The input is the text data selected by the user, and the output is an HTTP request to the server.

[1744] Step 6: Convert text to speech

[1745] server

[1746] The server sends the received selected text data to the speech generation API. Amazon Polly is used here. Authentication is performed using an API key, and a speech conversion request is sent specifying the required parameters (e.g., language, tone of voice). The server receives the speech data obtained from the speech generation API. The input is the selected text data, and the output is the generated speech data.

[1747] Step 7: Playing back audio data

[1748] Terminal

[1749] Receives audio data sent from the server. The device receives the audio data and plays it using an audio player (e.g. HTML5 Audio). JavaScript is used to set the URL of the audio data in the audio player, and the audio is played when the play button is pressed. The input is the received audio data, and the output is the audio played to the user.

[1750] (Application example 1)

[1751] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1752] Conventional food delivery applications lack effective means of conveying audio information, such as restaurant promotional videos and menu introductions, to users. This poses a challenge, making it difficult to provide sufficient information to users with hearing impairments or who have difficulty understanding foreign languages. Furthermore, the lack of information provision through audiovisual interaction raises concerns about a decline in the quality of the user experience.

[1753] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1754] In this invention, the server includes means for transmitting voice data to a voice analysis API and acquiring text data, means for formatting the acquired text data and assigning a timestamp, means for displaying the text data on a user interface, means for receiving a user's text selection and transmitting it to a voice generation API to acquire voice data, means for playing the voice data, and means for capturing and converting the voice of an advertising medium into text to provide food delivery information, and converting the text selected by the user into voice and playing it back. This allows the user to visually check the voice information converted into text in real time, and further select the text as needed and play it back as voice information.

[1755] "Information media" refers to means of providing information through audio and video. This includes television broadcasts, radio broadcasts, online videos, etc.

[1756] A "voice analysis API" is an application programming interface (API) for analyzing voice data and converting it into text data. It is generally implemented using a cloud-based service.

[1757] A "user interface" is an interface that allows a user to interact with a system. It includes means such as a screen display and audio output.

[1758] A "Speech Generation API" is an application programming interface (API) for converting text data into speech data. It provides text-to-speech functionality.

[1759] "Audio data" means a digital representation of sound waves, typically captured by a microphone or recorded audio.

[1760] "Text data" refers to data written in characters. This includes character strings generated from audio data using a voice analysis API.

[1761] "Food delivery information" refers to information about food and drink delivery, including restaurant promotional videos, menu introductions, delivery status, etc.

[1762] "Promotional media" refers to the means of widely disseminating product information. This includes videos, audio, posters, etc.

[1763] This invention is a system that converts audio from television broadcasts and other information media into text in real time and displays it on a user interface. It also has the function of converting user-selected text into audio and playing it back. The following are specific examples of how this system can be implemented.

[1764] System Configuration

[1765] The system consists of three main components: a server, a terminal, and a user.

[1766] server

[1767] The server operates a program that uses the voice analysis API and voice generation API, which provides the following functions:

[1768] Send the audio data to the audio analysis API and obtain the text data.

[1769] The acquired text data is formatted and a timestamp is added.

[1770] The formatted text data is sent to the terminal and displayed.

[1771] It receives a text selection request from the user, sends the text to the speech generation API, and obtains the speech data.

[1772] Terminal

[1773] A terminal is a device (e.g., a smartphone or tablet) that is operated by a user. A terminal performs the following functions:

[1774] Capture audio using the built-in microphone or an external device.

[1775] Send the captured audio data to the server.

[1776] The text data transmitted from the server is displayed on the user interface in real time.

[1777] The text data selected by the user is sent again to the server, and the voice data is acquired.

[1778] The acquired audio data is played back.

[1779] User

[1780] The User is the final user of the system and performs the following operations:

[1781] Watch television broadcasts and other information media and capture audio through your device.

[1782] View real-time text data displayed in the user interface.

[1783] Select a specific portion of text and make a speech conversion request.

[1784] Listen to the audio being played and understand the information.

[1785] Program processing procedure

[1786] The server first receives the voice data sent from the device. Next, it sends the voice data to a voice analysis API to obtain text data. This text data is formatted and a timestamp is added. The formatted text is sent to the device and displayed in the user interface. When the user selects a portion of the text and requests voice conversion, the server sends the text data to a voice generation API to obtain the voice data. Finally, the voice data is sent to the device and played.

[1787] Hardware and software used

[1788] Hardware

[1789] Smartphones, tablets, and microphones

[1790] software

[1791] Speech analysis API (e.g., Google Cloud Speech-to-Text)

[1792] Speech generation APIs (e.g., Google Text-to-Speech)

[1793] Server-side program (implemented in Python etc.)

[1794] User Interface (using a front-end framework)

[1795] Specific examples

[1796] When a user uses a food delivery app, the audio from a restaurant's promotional video is captured by the smartphone's microphone. This audio is converted into text in real time and displayed on the user interface. The user selects the part of the displayed text that says "Today's special is sushi pizza" and requests that it be played aloud. The server sends this text to a speech generation API, which then transmits the generated audio to the device. Finally, the user hears the audio saying "sushi pizza."

[1797] Prompt Sentence Examples

[1798] "When a user types "sushi pizza" into your application, convert the text into speech and play it back."

[1799] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1800] Output results according to processing steps

[1801] Step 1:

[1802] The device captures audio from information media using a built-in microphone or an external device, and this audio data is collected in real time and sent to a server for further processing.

[1803] Input: Audio data (audio from information media)

[1804] Output: Audio data is sent to the server

[1805] Step 2:

[1806] The server receives the voice data sent from the device, then uses a voice analysis API to convert the voice data into text data, which is then formatted and timestamped.

[1807] Input: Audio data

[1808] Output: Formatted text data (with timestamp)

[1809] Step 3:

[1810] The server sends the formatted text data to the terminal.

[1811] Input: Formatted text data

[1812] Output: Text data is sent to the terminal

[1813] Step 4:

[1814] The terminal displays the transmitted text data on the user interface in real time, allowing the user to check the text data displayed in a visible format.

[1815] Input: Text data

[1816] Output: Real-time displayed text data

[1817] Step 5:

[1818] The user selects the desired part from the text data, and the selected text is sent back to the server as a speech generation request.

[1819] Input: Real-time displayed text data

[1820] Output: The selected text data is sent to the server.

[1821] Step 6:

[1822] The server uses a speech generation API to convert the selected text data into speech data, which is then sent to the device.

[1823] Input: Selected text data

[1824] Output: Generated audio data

[1825] Step 7:

[1826] The terminal plays the audio data that has been sent, and the user can hear the audio corresponding to the selected text.

[1827] Input: Generated audio data

[1828] Output: Played audio data (user hears the audio)

[1829] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1830] Summary of the Invention

[1831] This invention is a system that converts television broadcast audio into text in real time and displays that text on a user interface. It also has the ability to convert user-selected text into speech and play it back. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, the user experience can be further improved. This system is realized using a speech analysis API and a speech generation API.

[1832] System Configuration

[1833] This system consists of three main components: a server, a terminal, and a user. The server is responsible for converting speech to text and generating speech, while the terminal is responsible for capturing speech, displaying text, playing speech, and analyzing emotions using an emotion engine. The user operates the system through the terminal.

[1834] Overview of program processing

[1835] 1. Capture and transmit audio

[1836] Terminal

[1837] While the user is watching a television program, the terminal captures the television sound in real time using a built-in microphone or a connected external device.

[1838] The captured audio data is periodically sent to a server.

[1839] 2. Speech-to-text

[1840] server

[1841] The server receives the voice data sent from the device and sends it to the voice analysis API.

[1842] The voice analysis API analyzes voice data and converts it into text data.

[1843] The server receives the converted text data and formats it.

[1844] 3. Real-time subtitle generation and display

[1845] Terminal

[1846] The formatted text data is sent to the terminal and displayed in real time on the user interface.

[1847] 4. User's voice conversion request

[1848] User

[1849] The user selects a particular portion of the displayed text and requests to hear that text aloud.

[1850] Terminal

[1851] This request is sent to the server along with the text data.

[1852] 5. Text-to-speech

[1853] server

[1854] The server sends the text data selected by the user to the speech generation API.

[1855] The speech generation API analyzes text data and converts it into speech data.

[1856] The server receives the generated audio data.

[1857] 6. Playing audio data

[1858] Terminal

[1859] The audio data is sent to the terminal and played back via an audio player.

[1860] 7. Emotion Analysis Using an Emotion Engine

[1861] Terminal

[1862] The device captures the user's facial expressions or tone of voice in real time and sends them to the emotion engine.

[1863] The emotion engine analyzes the user's emotions and returns the results to the terminal.

[1864] 8. Emotion-Based Interface Adjustment

[1865] Terminal

[1866] Based on the analysis results from the emotion engine, the content of the text data displayed and the tone and tempo of the audio data are adjusted.

[1867] Specific examples

[1868] Example 1: For a hearing impaired user

[1869] Voice input

[1870] A user is watching a news program. During this time, the device captures the audio of the news and periodically transmits it to a server.

[1871] Text

[1872] The server sends the audio data of the news program to a speech analysis API and obtains text data such as, "Tonight's news will report on important events from around the world."

[1873] Real-time subtitle display

[1874] The captured text data is sent to the terminal and displayed on the user interface, allowing the user to read the subtitles displayed on the screen.

[1875] Sentiment analysis and interface adjustment

[1876] The device sends the user's facial expressions and tone of voice to an emotion engine, which then adjusts the display content and the tone and tempo of the audio playback based on the analysis results.

[1877] Example 2: For foreign users who do not understand Japanese

[1878] Voice input

[1879] A user is watching a variety show. The device captures the audio of the variety show and sends it to the server.

[1880] Text

[1881] The server sends the audio data from the variety show to the audio analysis API and obtains text data such as, "This week's guest is a world-famous artist."

[1882] Voice conversion request

[1883] The user selects the subtitle portion, "This week's guest is a world-class artist," to hear it in English.

[1884] Voice generation

[1885] The server sends the selected text data to the speech generation API and obtains English speech data.

[1886] Audio playback and emotion analysis

[1887] The device plays back the acquired English audio data, while simultaneously analyzing the user's emotions using an emotion engine to adjust the tone and tempo of the audio playback.

[1888] The above is an example of a specific embodiment of the system according to the present invention, which aims to meet the diverse needs of users and make television viewing more accessible and personalized.

[1889] The processing flow will be explained below.

[1890] Step 1:

[1891] The device captures the audio of the TV program being watched by the user in real time using a built-in microphone or an external device, and stores the captured audio data in a buffer.

[1892] Step 2:

[1893] The terminal transmits the buffered audio data to the server at regular time intervals (for example, every second) using a protocol such as an HTTP POST request.

[1894] Step 3:

[1895] The server prepares to process the audio data received from the device and creates a request to send the audio data to the audio analysis API.

[1896] Step 4:

[1897] The server sends the voice data to the voice analysis API, which converts the voice data into text data.

[1898] Step 5:

[1899] The server receives the text data returned from the speech analysis API, formats the received text data, and adds timestamps, etc., as needed.

[1900] Step 6:

[1901] The server sends the formatted text data to the device using a WebSocket connection or an HTTP response.

[1902] Step 7:

[1903] The terminal acquires the text data received from the server and displays the acquired text data as real-time subtitles on the user interface.

[1904] Step 8:

[1905] The device captures the user's facial expressions or tone of voice in real time and sends them to the emotion engine, which analyzes the user's emotions and sends the results back to the device.

[1906] Step 9:

[1907] The device adjusts the displayed content of the text data and the tone and tempo of the voice based on the analysis results from the emotion engine. If the user's emotion is positive, the device brightens the displayed text and brightens the tone of the voice.

[1908] Step 10:

[1909] The user selects a specific portion of the displayed subtitles and clicks on the corresponding interface button when he or she wishes to hear the text spoken.

[1910] Step 11:

[1911] The terminal receives the user's selection operation and transmits the selected text data and a voice conversion request to the server.

[1912] Step 12:

[1913] The server creates a request to send the selected text data received from the device to the speech generation API.

[1914] Step 13:

[1915] The server sends the text data to the speech generation API and requests speech conversion, which converts the text data into speech data.

[1916] Step 14:

[1917] The server receives the voice data returned from the voice generation API and sends it to the device as is.

[1918] Step 15:

[1919] The terminal acquires the audio data received from the server and plays the acquired audio data using an audio player so that the user can hear it.

[1920] The above are the specific steps of the system's processing, and the specific operations at each step are described, allowing the system's operations to be clearly understood.

[1921] Example 2

[1922] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1923] In conventional television broadcast viewing systems, there are users who cannot properly receive audio information due to hearing impairments or language barriers. For this reason, there is a need for a method to convert audio into text in real time and display that text to the user. However, existing systems have difficulty providing a personalized interface that takes user emotions into account. In addition, there is also the challenge of seamlessly capturing audio data in real time, converting it into text, and playing back the text as audio.

[1924] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a voice analysis means, a user interface means, a voice generation means, an emotion analysis means, and an interface adjustment means. This makes it possible to convert voice into text in real time and provide the text to the user in a personalized format. In addition, the process from capturing voice data to converting it into text and then playing the voice can be seamlessly linked, improving the user experience.

[1925] The "voice analysis means" is a means for analyzing voice data and converting it into text data.

[1926] The "user interface means" is a means for displaying text data to a user in real time and providing an interface for the user to input data.

[1927] The "voice generating means" is a means for analyzing text data and converting it into voice data.

[1928] The "emotion analysis means" is a means for capturing the user's facial expressions and tone of voice, and analyzing the data to recognize emotions.

[1929] The "interface adjustment means" is a means for adjusting the user interface, tone, tempo, etc. of audio playback based on the analyzed emotions.

[1930] System Overview

[1931] The present invention is a system that converts television broadcast audio into text in real time and displays the text on a user interface. It also has the function of converting user-selected text into speech and playing it back. The system can improve the user experience by recognizing the user's emotions and adjusting the interface based on them. This system is realized using a speech analysis means and a speech generation means.

[1932] Hardware and software used

[1933] Hardware:

[1934] Client device (terminal): Captures TV audio and displays text and plays audio.

[1935] Server: Analyzes and generates audio data.

[1936] software:

[1937] Speech analysis API: Analyzes voice data and converts it into text data (for example, Google Cloud Speech-to-Text API).

[1938] Speech generation API: Analyzes text data and converts it into speech data (e.g., Amazon Polly).

[1939] Emotion analysis API: Analyzes a user's facial expressions and tone of voice to recognize their emotions (for example, Microsoft Azure Cognitive Services Emotion API).

[1940] Program processing

[1941] 1. Capture and transmit audio

[1942] Terminal

[1943] While a user is watching a TV program, the device captures the TV audio in real time using a built-in microphone or a connected external device, and the captured audio data is periodically sent to a server.

[1944] 2. Speech-to-text

[1945] server

[1946] The server receives the voice data sent from the device and sends it to the voice analysis API. The voice analysis API analyzes the voice data and converts it into text data. The server then formats the converted text data and makes any necessary corrections to fit the context.

[1947] 3. Real-time subtitle generation and display

[1948] Terminal

[1949] Receives formatted text data sent from the server and displays it in the user interface in real time. Adjusts the font size and color of the text so that the user can easily read the text data.

[1950] 4. User's voice conversion request

[1951] User

[1952] The user selects a particular portion of text displayed on the screen and makes a request to hear the selected text spoken.

[1953] Terminal

[1954] It receives a user request and sends the text data to the server.

[1955] 5. Text-to-speech

[1956] server

[1957] The server sends the text data selected by the user to the speech generation API. The speech generation API analyzes the text data and converts it into speech data. The server receives the generated speech data and sends it back to the device.

[1958] 6. Playing audio data

[1959] Terminal

[1960] The device decompresses the received audio data and plays it through an audio player, providing an interface that allows you to adjust the volume and speed of the audio being played.

[1961] 7. Emotion Analysis Using an Emotion Engine

[1962] Terminal

[1963] The device captures the user's facial expressions or tone of voice using a built-in camera or microphone, and sends the captured data to the emotion engine, which analyzes the user's emotions in real time and sends the results back to the device.

[1964] 8. Emotion-Based Interface Adjustment

[1965] Terminal

[1966] Based on the analysis results from the emotion engine, the content displayed in the user interface and the tone and tempo of the audio data can be adjusted. For example, if the user shows a confused expression, the text font can be made larger and the audio playback speed can be slowed down.

[1967] Specific examples and prompts for the generative AI model

[1968] Example 1: A hearing-impaired user

[1969] Voice input

[1970] A user is watching a news program. During this time, the device captures the audio of the news and periodically transmits it to a server.

[1971] Text

[1972] The server sends the audio data of the news program to the audio analysis API and obtains the text data, "Tonight's news will report on important events from around the world."

[1973] Real-time subtitle display

[1974] The captured text data is sent to the terminal and displayed on the user interface, allowing the user to read the subtitles displayed on the screen.

[1975] Sentiment analysis and interface adjustment

[1976] The device sends the user's facial expressions and tone of voice to an emotion engine, which then adjusts the display content and the tone and tempo of the audio playback based on the analysis results.

[1977] Example 2: For foreign users who do not understand Japanese

[1978] Voice input

[1979] A user is watching a variety show. The device captures the audio of the variety show and sends it to the server.

[1980] Text

[1981] The server sends the audio data from the variety show to the audio analysis API and obtains text data such as "This week's guest is a world-famous artist."

[1982] Voice conversion request

[1983] The user selects the subtitle portion, "This week's guest is a world-class artist," to hear it in English.

[1984] Voice generation

[1985] The server sends the selected text data to the speech generation API and obtains English speech data.

[1986] Audio playback and emotion analysis

[1987] The device plays back the acquired English audio data, while simultaneously analyzing the user's emotions using an emotion engine to adjust the tone and tempo of the audio playback.

[1988] Example prompts for generative AI models

[1989] 1. Example prompts for transcription of news programs

[1990] "Capture the audio of news programs and convert it into text data in real time using speech analysis methods and display it."

[1991] 2. Example prompts for sentiment analysis

[1992] "Capture the user's facial expressions and tone of voice, use sentiment analysis tools to perform sentiment analysis, and adjust the interface based on the results."

[1993] The above is an example of a specific embodiment of the system according to the present invention, which aims to meet the diverse needs of users and make television viewing more accessible and personalized.

[1994] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1995] Step 1:

[1996] Audio capture and transmission

[1997] Terminal

[1998] While the user is watching a TV program, the device captures the TV audio in real time using the built-in microphone or a connected external device. This inputs the TV audio into the device as audio data. The captured audio data is temporarily stored in buffer memory. The audio data is then compressed at regular intervals (e.g., every 5 seconds) and sent to the server.

[1999] Step 2:

[2000] Speech to text

[2001] server

[2002] The server receives the voice data sent from the terminal. It then sends the received voice data to a voice analysis means (e.g., a voice analysis API). The voice analysis means analyzes the voice data and converts it into text data. During this process, the voice signal is digitized and context estimation is performed in conjunction with phonemic analysis. After the converted text data is sent back to the server, the server formats the text data and corrects it into an easy-to-read format. This formatted text data is then output.

[2003] Step 3:

[2004] Real-time subtitle generation and display

[2005] Terminal

[2006] The formatted text data sent from the server is received. The received text data is displayed in real time on the user interface. When displayed, the size and color of the characters, the transparency of the background, etc. are adjusted to make it easy for the user to read. At this time, the text data is input into the user interface and output in a visually represented form.

[2007] Step 4:

[2008] User's voice conversion request

[2009] User

[2010] The user selects a specific portion of text displayed on the screen. For example, they select a segment of a news program: "Tonight's news will report on important events from around the world." This selection is entered into the device's interface. The user then makes a request to hear the selected text aloud. This request triggers the transmission of the selected text data from the device to the server.

[2011] Step 5:

[2012] Text-to-speech

[2013] server

[2014] The server receives the text data selected by the user. This text data is sent to a speech generation means (e.g., a speech generation API). The speech generation means analyzes the text data and converts it into speech data. Specifically, it performs text analysis and generates a speech file using a speech synthesis engine. This generated speech data is sent back to the server, and then retransmitted from the server to the terminal.

[2015] Step 6:

[2016] Playing audio data

[2017] Terminal

[2018] The device decompresses the audio data received from the server and plays it through the built-in audio player. This audio data is played in an audio format (e.g., MP3) and output to the user's ears. The user has an interface to adjust the volume and speed of the audio being played.

[2019] Step 7:

[2020] Emotion analysis using an emotion engine

[2021] Terminal

[2022] The device uses a built-in camera and microphone to capture the user's facial expressions and tone of voice. This captured data is sent to an emotion analysis means (e.g., emotion analysis API). The emotion analysis means analyzes this data and identifies the user's emotions (e.g., joy, sadness, anger, confusion, etc.). The analysis results are sent back to the device in real time and output as the user's emotion data.

[2023] Step 8:

[2024] Emotion-based interface adjustment

[2025] Terminal

[2026] Based on the analysis results received from the emotion analysis means, the device adjusts the content displayed in the user interface and the tone and tempo of the audio data. For example, if the user shows a confused expression, the device increases the font size of the text and slows down the audio playback speed. This series of adjustment operations is reflected in the user interface and is output as an optimized interface.

[2027] This is the specific flow of the program processing for this system. The inputs and outputs at each processing step, as well as the specific operations, are clearly shown, allowing for a more detailed understanding of the operation of the entire system.

[2028] (Application example 2)

[2029] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2030] By converting television broadcast audio into text in real time and displaying it on a user interface, users with hearing impairments or difficulty understanding foreign languages ​​can easily understand the broadcast content. However, this system faces several challenges. For example, if a user selects a specific text section and wants to hear it aloud, fast and accurate speech conversion is required. Furthermore, to more individually optimize the user experience, it is necessary to analyze the user's emotions and state in real time and adjust the interface and playback speed accordingly.

[2031] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for capturing audio from a television broadcast, means for transmitting the captured audio data to a voice analysis API, means for displaying text data acquired from the voice analysis API on a user interface in real time, means for receiving a user's text selection and transmitting the selected text data to a voice generation API, means for playing back the audio data acquired from the voice generation API, means for capturing the user's facial expression or tone of voice and transmitting it to an emotion analysis engine, and means for adjusting the interface and playback speed based on the analysis results of the emotion analysis engine. This not only meets the diverse needs of users, but also makes it possible to optimize the user experience in a more personalized manner.

[2032] "Television broadcasting" is a broadcasting format that transmits images and sounds to ordinary households or specific regions using radio waves or cables.

[2033] "Audio capture" is the process of collecting audio data in digital form using a microphone or other audio input device.

[2034] "Audio data" is a digital representation of captured audio in a data format that can be analyzed and processed.

[2035] A "voice analysis API" is a programming interface for analyzing voice data and converting it into text data.

[2036] "Text data" is information stored in the form of character string data, and is used for displaying on a user interface or converting into voice.

[2037] "User interface" is a general term for the screens and input / output devices that allow users to interact with a system.

[2038] A "voice generation API" is a programming interface for converting text data into voice data.

[2039] "Audio data playback" is the process of audibly outputting digital audio data through an audio playback device.

[2040] An "emotion analysis engine" is a software component that analyzes a user's facial expressions and tone of voice to recognize their emotional state.

[2041] "Interface adjustment" is the process of changing the parameters of the user interface and audio playback based on the analysis results of the emotion analysis engine.

[2042] "Time-stamped text" refers to text data to which specific time information has been added, and corresponds to a specific portion of audio content.

[2043] MODE FOR CARRYING OUT THE INVENTION

[2044] Summary of the Invention Embodiments

[2045] The present invention relates to a system that converts television broadcast audio into text in real time and displays the text on a user interface. This system analyzes audio data captured during television viewing and generates text data. Furthermore, the system has a function that allows a user to select a specific text portion, converts the selected text into audio, and plays it back. Furthermore, by combining it with an emotion analysis engine, the system can adjust the interface and audio playback based on the user's emotions.

[2046] System Configuration

[2047] This system consists of three main components: a server, a terminal, and a user.

[2048] server:

[2049] Use a speech analysis API to convert audio data into text.

[2050] Uses the speech generation API to convert the selected text into audio data.

[2051] Device:

[2052] It captures audio data from television broadcasts in real time and displays the text.

[2053] It receives the user's voice requests and sends them to the server.

[2054] It uses a sentiment analysis engine to analyze user emotions and adjust the interface and audio playback accordingly.

[2055] User:

[2056] Watch a TV broadcast and select the displayed text to request a voice conversion.

[2057] Emotions are input through facial expressions and tone of voice.

[2058] Program processing

[2059] Audio capture and transmission

[2060] The device captures audio from a television broadcast using a built-in or external microphone and periodically transmits the captured audio to a server. For example, while watching a news program, the device continuously captures the audio of the news.

[2061] Speech to text

[2062] The server sends the captured audio data to a speech analysis API (such as the Google Speech API) and converts it into text data. This process generates text data such as "Tonight's news will report on important events from around the world."

[2063] Real-time text display

[2064] The generated text data is sent from the server to the terminal and displayed in real time on the user interface, allowing the user to visually confirm the news anchor's remarks as text in real time.

[2065] User's voice conversion request

[2066] The user selects a specific text segment and sends a request to the server via the terminal to listen to it aloud. For example, if the user selects "This week's guest is a world-famous artist," a voice conversion request for that segment is sent.

[2067] Text-to-speech

[2068] The server sends the selected text data to a speech generation API (e.g., Amazon Polly) to generate corresponding voice data, which is then sent back to the device.

[2069] Playing audio data

[2070] The device plays the received audio data through an audio player. In this case, after selecting the text "This week's guest is a world-famous artist," the user can listen to the content in English.

[2071] Analysis by sentiment analysis engine

[2072] The device captures the user's facial expressions and tone of voice and sends them to an emotion analysis engine (using OpenCV and FER, for example). Based on the analysis results, the device adjusts the user interface, audio playback speed, etc.

[2073] Prompt Sentence Examples

[2074] "Demonstrate an application that translates captured text in Japanese in real time and adjusts the interface and voice playback based on the user's emotions."

[2075] This makes it possible to provide a system that meets the diverse needs of users and makes television viewing more accessible and personalized.

[2076] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2077] Program processing flow

[2078] Step 1:

[2079] Audio capture and transmission

[2080] The device uses a built-in or external microphone to capture the audio of the TV broadcast in real time. It then periodically transmits the captured audio data to a server. The input is the audio of the TV broadcast, and the output is digital audio data. Specifically, the device's microphone is activated, the audio data is temporarily stored in a buffer, and the buffer is periodically uploaded to the server.

[2081] Step 2:

[2082] Speech to text

[2083] The server sends the received voice data to the voice analysis API. The voice analysis API converts the voice data into text data and sends the text data back to the server. The input is voice data and the output is text data. Specifically, the server calls the voice analysis API, sends the voice data, and receives the text data returned from the API.

[2084] Step 3:

[2085] Real-time text display

[2086] Text data sent from the server is sent to the terminal and displayed in real time on the user interface. The input is text data from the server, and the output is the text displayed on the user interface. Specifically, the terminal receives the text data and immediately reflects it in the display area of ​​the user interface.

[2087] Step 4:

[2088] User's voice conversion request

[2089] The user selects a specific portion of the displayed text and requests to hear the selected text aloud. The input is the selected text on the user interface, and the output is the requested data. Specific actions require the user to select the text using a touch screen or mouse and press a request button.

[2090] Step 5:

[2091] Text-to-speech

[2092] Text data requested from the device is sent to the server, which then uses a speech generation API to convert the text data into voice data. The voice data returned from the voice generation API is stored on the server and sent to the device. The input is the selected text data, and the output is voice data. Specifically, the server calls the voice generation API, sends the text data, and receives the generated voice data.

[2093] Step 6:

[2094] Playing audio data

[2095] The device plays the received audio data through an audio player. The input is the audio data from the server, and the output is the played audio. Specifically, the device starts the audio player, decodes the received audio data, and plays it.

[2096] Step 7:

[2097] Analysis by sentiment analysis engine

[2098] The device captures the user's facial expressions and tone of voice in real time and sends them to an emotion analysis engine. The emotion analysis engine analyzes the user's emotions and sends the results back to the device. The input is the captured video or audio data, and the output is the analyzed emotion data. Specific operations include activating the device's camera, capturing the user's facial expressions, and sending the video data to the emotion analysis engine.

[2099] Step 8:

[2100] Emotion-based interface adjustment

[2101] Based on the analysis results from the emotion analysis engine, the device adjusts the tone and tempo of the user interface and audio playback. The input is emotion analysis data, and the output is adjusted interface elements and audio playback. Specifically, the device automatically changes the color scheme of the user interface and adjusts the playback speed according to the analysis results.

[2102] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[2103] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2104] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[2105] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[2106] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[2107] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[2108] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[2109] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[2110] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[2111] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[2112] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[2113] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[2114] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[2115] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[2116] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[2117] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[2118] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[2119] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[2120] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[2121] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[2122] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[2123] The following is further disclosed regarding the above embodiment.

[2124] (Claim 1)

[2125] means for capturing audio from a television broadcast;

[2126] a means for transmitting the captured audio data to a speech analysis API;

[2127] a means for displaying text data obtained from the speech analysis API in a user interface in real time;

[2128] means for receiving a user's text selection and transmitting the selected text data to the speech generation API;

[2129] A system including means for playing audio data obtained from an audio generation API.

[2130] (Claim 2)

[2131] 2. The system according to claim 1, wherein the text generating means formats the acquired text data and adds a timestamp.

[2132] (Claim 3)

[2133] 10. The system of claim 1, further comprising an interface for allowing a user to select a particular portion of the text data to be displayed in real time.

[2134] "Example 1"

[2135] (Claim 1)

[2136] means for capturing audio from a television broadcast;

[2137] means for transmitting the captured audio data to a server;

[2138] A means for the server to send the voice data to a voice analysis API and format the text data obtained from the voice analysis API;

[2139] means for displaying the formatted text data in a user interface in real time;

[2140] means for receiving a user's text selection and transmitting the selected text data to the speech generation API;

[2141] A system including means for playing audio data obtained from an audio generation API.

[2142] (Claim 2)

[2143] 2. The system according to claim 1, wherein the audio capture means acquires audio data using a built-in microphone or an external device, and periodically transmits the audio data to the server.

[2144] (Claim 3)

[2145] 2. The system according to claim 1, further comprising means for the server to send voice data to the voice analysis API, format text data obtained from the voice analysis API, and assign a timestamp.

[2146] "Application Example 1"

[2147] (Claim 1)

[2148] a means for capturing audio of the information media;

[2149] a means for transmitting the captured audio data to a speech analysis API;

[2150] a means for displaying text data obtained from the speech analysis API in a user interface in real time;

[2151] means for receiving a user's text selection and transmitting the selected text data to the speech generation API;

[2152] a means for playing back the audio data obtained from the audio generation API;

[2153] The system includes a means for capturing audio from advertising media to provide food delivery information, converting it into text, and displaying it, and converting text selected by a user into audio and playing it back.

[2154] (Claim 2)

[2155] 2. The system according to claim 1, wherein the text generating means formats the acquired text data and adds a timestamp.

[2156] (Claim 3)

[2157] 10. The system of claim 1, further comprising an interface for allowing a user to select a particular portion of the text data to be displayed in real time.

[2158] "Example 2: Combining Emotion Engines"

[2159] (Claim 1)

[2160] means for capturing audio from a television broadcast;

[2161] means for transmitting the captured voice data to a voice analysis means;

[2162] means for displaying the text data acquired from the voice analysis means on a user interface in real time;

[2163] means for receiving a user's text selection and transmitting the selected text data to the speech generating means;

[2164] means for reproducing the audio data acquired from the audio generating means;

[2165] A means of capturing the user's facial expressions and tone of voice to analyze their emotions;

[2166] The system includes means for adjusting a user interface or audio data based on the analyzed emotions.

[2167] (Claim 2)

[2168] 10. The system of claim 1, wherein the acquired text data is formatted and time-stamped.

[2169] (Claim 3)

[2170] 10. The system of claim 1, further comprising an interface for allowing a user to select a particular portion of the text data to be displayed in real time.

[2171] "Application example 2 when combining emotion engines"

[2172] (Claim 1)

[2173] means for capturing audio from a television broadcast;

[2174] a means for transmitting the captured audio data to a speech analysis API;

[2175] a means for displaying text data obtained from the speech analysis API in a user interface in real time;

[2176] means for receiving a user's text selection and transmitting the selected text data to the speech generation API;

[2177] a means for playing back the audio data obtained from the audio generation API;

[2178] means for capturing and transmitting a user's facial expression or tone of voice to an emotion analysis engine;

[2179] A system including a means for adjusting an interface or playback speed based on the analysis results of an emotion analysis engine.

[2180] (Claim 2)

[2181] 2. The system according to claim 1, wherein the text generating means formats the acquired text data and adds a timestamp.

[2182] (Claim 3)

[2183] 10. The system of claim 1, further comprising an interface for allowing a user to select a particular portion of the text data to be displayed in real time. [Explanation of symbols]

[2184] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. means for capturing audio from a television broadcast; a means for transmitting the captured audio data to a speech analysis API; a means for displaying text data obtained from the speech analysis API in a user interface in real time; means for receiving a user's text selection and transmitting the selected text data to the speech generation API; A system including means for playing audio data obtained from an audio generation API.

2. 2. The system according to claim 1, wherein the text generating means formats the acquired text data and adds a timestamp.

3. 10. The system of claim 1, further comprising an interface for allowing a user to select a particular portion of the text data to be displayed in real time.

Images