System

Abstract

A system is provided.SOLUTION: The system includes a means for collecting the voice of a lecturer, a means for converting the collected voice into text data, a means for analyzing the context of the text data and selecting an appropriate response, a means for transmitting the selected response as a chat message, and a means for displaying the chat message on a screen.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In online study groups and lectures, there are challenges such as silence during Q&A sessions and lectures, and speakers often feel anxious due to a lack of participant response. For this reason, it is necessary to stimulate participants' interest and promote two-way communication between speakers and participants. [Means for solving the problem]

[0005] The present invention is a system that includes means for collecting a speaker's voice and converting the collected voice into text data, means for analyzing the context of the text data and selecting appropriate backchannels, means for transmitting the selected backchannels as chat messages, and means for displaying the chat messages on a screen. This allows participants to receive appropriate backchannels in response to the lecture content, and the speaker can check the participants' reactions in real time. Furthermore, by using speech recognition technology and natural language processing technology, it is possible to generate more accurate and appropriate backchannels. As a result, the interactivity of online study groups and lectures is improved, promoting communication and stimulating participant interest.

[0006] 1. "Means for collecting the speaker's voice" means a device or program that collects the speaker's voice in real time.

[0007] 2. "Means for converting collected voice data into text data" means a technology or device that analyzes collected voice data and converts it into corresponding text data.

[0008] 3. "Means for analyzing the context of text data and selecting appropriate backchannels" refers to a system that uses natural language processing technology to analyze the obtained text data and determine and select backchannels that fit the context.

[0009] 4. "Means for sending selected backchannels as chat messages" means a technology or program that generates selected backchannels in text format and sends the message to an online chat platform.

[0010] 5. "Means for displaying chat messages on a screen" means a device or application that displays sent chat messages on a user's screen.

[0011] 6. "Speech recognition technology" is a technology that analyzes voice data and converts that voice into text.

[0012] 7. "Natural language processing technology" is a technology that analyzes text data, understands its context, and generates appropriate responses and actions. [Brief explanation of the drawings]

[0013] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0014] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0015] First, the terms used in the following description will be explained.

[0016] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0017] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0018] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0019] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0020] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0021] [First embodiment]

[0022] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0023] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0024] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0025] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0026] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0027] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0028] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0029] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0030] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0031] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0032] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0033] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0034] This invention is a system for realizing interactive communication between speakers and participants in online study groups and lectures. Specifically, it collects the speaker's voice, converts that voice into text, analyzes the context of the text, and automatically generates and transmits appropriate responses.

[0035] This system mainly consists of the following devices and technologies:

[0036] 1. Collecting speaker audio

[0037] The user (speaker) uses a device with a microphone (computer, smartphone, tablet, etc.) to give a lecture or explanation.

[0038] The device collects the speaker's voice in real time and transmits it to the server as digital audio data.

[0039] 2. Converting voice data to text

[0040] The server converts the received voice data into text data using voice recognition technology, using a voice recognition engine (such as Google (registered trademark) Speech-to-Text API).

[0041] The server receives the converted text data and prepares it for contextual analysis.

[0042] 3. Context Analysis and Backchannel Generation

[0043] The server uses natural language processing technology to analyze the context of the text data, identifying what the speaker said and when, and then selects appropriate back-channel responses (e.g., "Uh-huh," "I see," "Great!", "Amazing!", etc.).

[0044] The server automatically generates the selected responses in the form of chat messages.

[0045] 4. Sending and Viewing Chat Messages

[0046] The server transmits the generated backchannel message to the terminal.

[0047] The terminal displays the sent chat message on the user's screen so that all participants can see the message.

[0048] Specific examples

[0049] Example 1: A speaker is explaining a theory

[0050] 1. Audio collection

[0051] User: "Based on this theory, performance will improve significantly."

[0052] Device: Collects audio from the microphone and sends it to the server.

[0053] 2. Text Conversion

[0054] Server: Sends the speech data to the speech recognition engine and receives the text data, "Based on this theory, performance will be significantly improved."

[0055] 3. Context Analysis and Backchannel Generation

[0056] Server: Analyzes the text data using natural language processing technology and selects the appropriate "Like!" response phrase based on the context.

[0057] Server: Generate a "Like" chat message.

[0058] 4. Chat sending and display

[0059] Server: Sends the generated "like" chat message to the device.

[0060] Terminal: Chat messages are displayed on the user's screen.

[0061] In this way, appropriate responses are automatically inserted while the speaker is speaking, making it easier for participants to actively engage with the content of the lecture, and the speaker can also see participants' reactions in real time. This system improves the quality of online communication and provides a more effective learning environment.

[0062] The processing flow will be explained below.

[0063] Step 1:

[0064] The user accesses the online study group link and enters their login information.

[0065] The terminal sends the entered login information to the server.

[0066] The server checks the authentication information against a database to verify that the participant is legitimate.

[0067] Step 2:

[0068] The user (speaker) uses a device with a microphone to begin a lecture or explanation.

[0069] The device collects the speaker's audio in real time.

[0070] The device transmits the collected voice data to the server in real time.

[0071] Step 3:

[0072] The server sends the received voice data to the voice recognition engine.

[0073] A voice recognition engine converts the voice data into text data.

[0074] The server receives the converted text data.

[0075] Step 4:

[0076] The server passes the text data to a natural language processing (NLP) module.

[0077] The NLP module analyzes the context of the text data and determines appropriate responses.

[0078] The server selects appropriate response phrases based on the analysis results.

[0079] Step 5:

[0080] The server generates the selected backchannel as a chat message.

[0081] The server sends the generated chat message to the device.

[0082] The terminal displays the chat messages on the user's screen.

[0083] Step 6:

[0084] Users (participants) check the chat messages and enter their reactions or questions in the chat field as necessary.

[0085] The device sends the input responses and questions to the server.

[0086] The server collects participants' responses and questions and uses them to generate subsequent responses and analyze context.

[0087] Example 1

[0088] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0089] In online study groups and lectures, there is a tendency for there to be a lack of interaction between the speaker and the participants, which can lead to a decline in the quality of communication. This makes it difficult for the speaker to get participants' reactions in real time, and for participants to maintain interest in the content of the lecture.

[0090] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0091] In this invention, the server includes means for collecting the speaker's voice, means for converting the collected voice into text data, means for analyzing the context of the text data and selecting appropriate backchannels, means for transmitting the selected backchannels as chat messages, and means for displaying the chat messages on a screen. This allows the speaker to receive appropriate backchannels from the participants in real time, and allows the participants to maintain their interest in the content of the lecture.

[0092] "Means for collecting the speaker's voice" refers to a device or function that uses the microphone of the device (computer, smartphone, tablet, etc.) used by the speaker to collect the voice during the lecture as digital voice data in real time.

[0093] "Means for converting collected voice data into text data" refers to technology that uses a voice recognition engine to convert collected digital voice data into text data. For example, Google Speech-to-Text API is one such technology.

[0094] "Means for analyzing the context of text data and selecting appropriate backchannels" refers to technologies and algorithms for analyzing the content and context of converted text data using natural language processing technology and selecting appropriate backchannel phrases.

[0095] "Means for transmitting the selected backchannels as chat messages" refers to technologies and protocols for converting the selected backchannel phrases into chat message format and transmitting them to participants' devices. Examples include WebSocket and HTTP.

[0096] "Means for displaying chat messages on a screen" refers to software and technology for displaying chat messages sent to participants' devices on an interface in real time. For example, this includes online conference tools and dedicated applications.

[0097] "Speech recognition technology" refers to technology for converting voice data into text data, including voice models and machine learning algorithms. An example is the Google Speech-to-Text API.

[0098] "Natural language processing technology" is a technology for analyzing the content and context of text data and understanding its meaning, and includes technologies such as morphological analysis, semantic analysis, and sentiment analysis. Models such as BERT are also part of this technology.

[0099] This invention is a system for realizing interactive communication between speakers and participants in online study groups and lectures. This system collects the speaker's voice, converts the voice into text, analyzes the context of the text, and automatically generates and transmits appropriate responses. The specific configuration and implementation method of this system are described in detail below.

[0100] Hardware and Software Use

[0101] 1. Audio collection

[0102] The user (speaker) gives a lecture or explanation using a device with a microphone (computer, smartphone, tablet, etc.).

[0103] The device uses a microphone to collect the speaker's voice in real time and uses an internet connection to send it to a server as digital audio data.

[0104] 2. Text Conversion

[0105] The server converts the received voice data into text data using speech recognition technology, using a speech recognition engine such as the Google Speech-to-Text API.

[0106] The server receives the converted text data and prepares it for contextual analysis.

[0107] 3. Context Analysis and Backchannel Generation

[0108] The server analyzes the context of the text data using natural language processing (NLP) techniques, including morphological analysis, semantic analysis, and sentiment analysis, and also uses AI models such as BERT.

[0109] The server selects appropriate backchannel phrases based on the analysis results. The selected backchannels are either chosen from a predefined list of phrases or dynamically generated by AI.

[0110] 4. Sending and Viewing Chat Messages

[0111] The server sends the generated backchannel message to the device using a communication protocol such as WebSocket or HTTP.

[0112] The device displays the sent chat messages on the user's screen using an online conference tool or a dedicated application.

[0113] Specific examples

[0114] For example, here is the flow when a speaker explains, "Based on this theory, performance will improve significantly."

[0115] 1. Audio collection

[0116] User: "Based on this theory, performance will improve significantly."

[0117] Device: Collects audio from the microphone and sends it to the server.

[0118] 2. Text Conversion

[0119] Server: Sends audio data to the Google Speech-to-Text API and receives text data that says, "Based on this theory, performance will be significantly improved."

[0120] 3. Context Analysis and Backchannel Generation

[0121] Server: Analyzes the text data using natural language processing technology and selects the appropriate "Like!" response phrase based on the context.

[0122] Server: Generate a "Like" chat message.

[0123] 4. Sending and Viewing Chat Messages

[0124] Server: Sends the generated "like" chat message to the device.

[0125] Terminal: A web application displays chat messages on the user's screen.

[0126] This system allows users to receive appropriate responses from participants in real time during a lecture, and also helps participants to become more interested in the content of the lecture, thereby improving the quality of online communication and providing a more effective learning environment.

[0127] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0128] Step 1: Audio Collection

[0129] Input: A user speaks using the microphone on their device (computer, smartphone, tablet, etc.).

[0130] What happens: The speaker gives a presentation or talk, and the microphone collects the audio.

[0131] Device operation: The device's microphone collects the speaker's voice in real time.

[0132] Output: Collected digital audio data is generated and used in the next processing step.

[0133] Step 2: Sending audio data

[0134] Input: Digital audio data collected by the device.

[0135] What happens next: Your device sends audio data to a server over your internet connection.

[0136] Device operation: The device prepares the collected digital audio data and initiates communication to send it to the server.

[0137] Output: Audio data is sent to and received by the server.

[0138] Step 3: Convert audio data to text

[0139] Input: Digital audio data received by the server.

[0140] Specific actions taken: The server uses speech recognition technology (e.g., Google Speech-to-Text API) to convert the voice data into text data.

[0141] Server operation: The server sends the voice data to the speech recognition engine, which analyzes the voice and generates text data.

[0142] Output: The audio data is converted into "text data" and used in the next context analysis step.

[0143] Step 4: Contextual analysis of text data

[0144] Input: Text data returned from the speech recognition engine.

[0145] Specific actions taken: The server uses natural language processing techniques to analyze the context of the text data, including morphological analysis, semantic analysis, and sentiment analysis.

[0146] Server operation: The server analyzes the text data, performs calculations to identify the content and timing of the speaker's remarks, and extracts key keywords and sentiment.

[0147] Output: Information based on the contextual analysis is generated and used in the next step of backchannel generation.

[0148] Step 5: Selecting and generating backchannel phrases

[0149] Input: Information based on contextual analysis.

[0150] What happens: The server selects appropriate backchannels based on the contextual analysis results. The selected backchannels are either chosen from a pre-defined list or dynamically generated by a generative AI model.

[0151] Server operation: The server selects appropriate backchannel phrases based on the analysis results and automatically generates them in the form of chat messages.

[0152] Output: The selected backchannel phrase is generated as a chat message and used in the next sending step.

[0153] Step 6: Send a chat message

[0154] Input: An automatically generated chat backchannel message.

[0155] Specific actions taken: The server sends chat messages over the Internet to the user's device.

[0156] Server operation: The server initiates communication to send the generated chat message to the specified device. WebSocket or HTTP is used as the communication protocol.

[0157] Output: The chat message is sent to the device.

[0158] Step 7: View chat messages

[0159] Input: The chat message sent by the server.

[0160] Specific actions taken: Display chat messages received by the device on the screen.

[0161] Terminal Action: Displays a new message in a chat message window or similar, allowing all participants to see the message in real time.

[0162] Output: A backchannel message will be displayed in the chat window.

[0163] This series of processes allows users to receive appropriate responses from participants in real time during a lecture, making it easier for participants to become interested in the content of the lecture. This improves the quality of online communication and provides a more effective learning environment.

[0164] (Application example 1)

[0165] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0166] In online shopping virtual stores, customers often have difficulty getting a proper response in real time when they have questions or comments about products, which can lead to a poor customer experience and reduced purchasing intent.

[0167] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0168] In this invention, the server includes means for collecting the speaker's voice, means for converting the collected voice into text data, means for analyzing the context of the text data and selecting appropriate backchannels, means for sending the selected backchannels as chat messages, means for displaying the chat messages on a screen, means for collecting customer voice, converting it into text, analyzing the context and generating a response, and means for displaying the generated response message on a display device, thereby improving interaction with customers in real time and enabling a better shopping experience.

[0169] The "means for collecting the speaker's voice" refers to a device or method for collecting the speaker's voice in real time using a microphone device or the like and converting it into digital voice data.

[0170] The "means for converting collected voice data into text data" refers to a device or method for converting collected voice data into text information using voice recognition technology.

[0171] "Means for analyzing the context of text data and selecting appropriate responses" refers to a device or method that uses natural language processing technology to analyze the content and timing of statements contained in text data, and generates or selects appropriate responses or comments based on that context.

[0172] The "means for transmitting the selected backchannel as a chat message" refers to a device or method for transmitting the selected backchannel or response message in a chat format via a network.

[0173] The "means for displaying chat messages on a screen" refers to a device or method for displaying received chat messages on a user interface so that the user can check their contents.

[0174] "Means for collecting customer speech, converting it into text, analyzing the context, and generating a response" refers to a device or method that collects speech uttered by a customer, converts it into text information, analyzes the context, and automatically generates an appropriate response.

[0175] "Means for displaying the generated response message on a display device" refers to a device or method that outputs the generated response message to a display or other display device so that the user can visually confirm the message.

[0176] This invention is a system that automatically supports interactive communication in online lectures and virtual stores using speech recognition and context analysis. This system collects user speech, converts it into text in real time, analyzes the context, generates appropriate responses, and displays them on a display device. To function effectively, this system requires the following hardware and software:

[0177] Hardware and Software

[0178] Microphone devices: Used to collect audio data. This includes computers, smartphones, tablets, etc. with microphones.

[0179] Computer server: Receives the voice data and analyzes it using speech recognition and natural language processing technologies. It converts the voice data into text using Google Cloud's Speech-to-Text API and applies the GPT-3 (registered trademark) .5-turbo model using the transformers library.

[0180] Display device: A display device that shows chat messages and reply messages to users.

[0181] Data processing flow

[0182] 1. Audio collection: The device (user's smartphone or computer) collects the user's audio through a microphone device and transmits it to the server as digital audio data.

[0183] 2. Converting speech data to text: The server converts the speech data received into text using Google Cloud's Speech-to-Text API and prepares it for contextual analysis.

[0184] 3. Contextual analysis and response generation: The server uses natural language processing techniques to analyze the context of the text data, and uses the GPT-3.5-turbo model to generate appropriate responses based on that analysis.

[0185] 4. Creating and displaying a chat message or response message: The server sends the generated response to the terminal, which displays the message on its display device.

[0186] Specific examples

[0187] Example 1: Virtual store example

[0188] 1. Audio collection:

[0189] Customer: "Do you have this jacket in any other colors?"

[0190] Device: Collects audio from the microphone and sends it to the server.

[0191] 2. Text conversion:

[0192] Server: Sends the voice data to the voice recognition engine and receives the text data "Do you have this jacket in other colors?"

[0193] 3. Context analysis and response generation:

[0194] Server: Analyzes the text data using natural language processing technology and generates a response based on the context: "Of course we have it. Currently, it comes in three colors: black, blue, and red."

[0195] 4. Chat sending and display:

[0196] Server: Sends the generated response message to the terminal.

[0197] Terminal: Displays the response message on the user's screen.

[0198] Example prompt sentence:

[0199] Customer: Do you have this jacket in other colors?

[0200] context:

[0201] Generate a response.

[0202] The present invention improves customer experience in virtual stores and enables real-time interactive communication.

[0203] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0204] Step 1:

[0205] Audio collection: The device (user's smartphone or computer) collects the user's voice using a microphone device. When the user speaks, the microphone captures it and converts it into digital audio data in real time. This digital audio data is sent to the next processing step.

[0206] Input: User's voice

[0207] Output: Digital audio data

[0208] What it does: The microphone captures audio and the application converts it into a digital format.

[0209] Step 2:

[0210] Speech-to-text conversion: The device sends the collected digital speech data to a server, which converts the speech to text using Google Cloud's Speech-to-Text API and prepares the converted text for analysis.

[0211] Input: Digital audio data

[0212] Output: Text data

[0213] What happens: The server sends the audio data to the Google Cloud Speech-to-Text API, which converts the audio into text.

[0214] Step 3:

[0215] Contextual analysis and response generation: The server receives the text data and analyzes its context using natural language processing techniques. This analysis uses the transformers library and the GPT-3.5-turbo model. The generative AI model generates an appropriate response prompt from the analysis results.

[0216] Input: Text data

[0217] Output: Response message

[0218] Specific operation: The server sends a prompt to the GPT-3.5-turbo model using text data as input, and the model generates a response message.

[0219] Step 4:

[0220] Sending a response message: The server sends the generated response message to the terminal, which receives it and prepares it for display on the display device.

[0221] Input: Response message

[0222] Output: Response message sent to the terminal

[0223] Specific operation: The server sends a response message to the terminal over the network.

[0224] Step 5:

[0225] Display: The terminal displays the received response message on a display device, allowing the user to see the generated response in real time.

[0226] Input: Response message sent to the terminal

[0227] Output: Response message displayed on display device

[0228] Specific actions: The terminal renders the response message on the screen and displays it on the display device.

[0229] This enables real-time two-way communication between users and the system, improving the customer experience in the virtual store.

[0230] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0231] This invention is a system for promoting interactive communication between speakers and participants in online study groups and lectures. In particular, it combines an emotion engine to analyze the user's emotions and generate and display appropriate responses, enabling more lively communication.

[0232] This system is composed of the following major devices and technologies:

[0233] 1. Collecting speaker audio

[0234] The user (speaker) begins a lecture or explanation using a device with a microphone.

[0235] The terminal collects the speaker's voice in real time and transmits it to the server as digital voice data.

[0236] 2. Converting voice data to text

[0237] The server transmits the received voice data to a voice recognition engine, which converts the voice data into text data.

[0238] The speech recognition engine returns the converted text data to the server, which receives it.

[0239] 3. Context Analysis and Backchannel Generation

[0240] The server passes the text data to a natural language processing (NLP) module for contextual analysis.

[0241] The NLP module analyzes the context of the text data and determines appropriate interjections (e.g., "Uh-huh," "I see," "Great!", "Amazing!", etc.).

[0242] The server selects and generates appropriate responses based on the analysis results.

[0243] 4. Sentiment analysis and backchannel optimization

[0244] The server passes the collected voice data and text data to the emotion engine.

[0245] The emotion engine analyzes emotions from the user's voice and text, and selects appropriate responses based on the results.

[0246] For example, through emotion analysis, if a speaker is speaking enthusiastically, the system will generate many positive responses such as "Amazing!", but if the listeners are feeling stressed, it will generate neutral or empathetic responses such as "I see" or "Did you understand?"

[0247] 5. Sending and Viewing Chat Messages

[0248] The server sends the generated chat message to the device.

[0249] The terminal displays the sent chat message on the user's screen so that all participants can see the message.

[0250] Specific examples

[0251] Example 1: A speaker is explaining a new theory

[0252] 1. Audio collection

[0253] User (speaker): "Based on this theory, performance will improve significantly."

[0254] Device: Collects audio from the microphone and sends it to the server.

[0255] 2. Text Conversion

[0256] Server: Sends the speech data to the speech recognition engine and receives the text data, "Based on this theory, performance will be significantly improved."

[0257] 3. Context Analysis and Backchannel Generation

[0258] Server: Analyzes the text data using natural language processing technology and selects a "Like!" response based on the context.

[0259] Server: Generate a "Like" chat message.

[0260] 4. Sentiment analysis and backchannel optimization

[0261] Server: Passes voice and text data to the emotion engine.

[0262] Emotion Engine: If a speaker is speaking confidently, it analyzes positive emotions and selects additional back-channel responses like "Awesome!" to emphasize them.

[0263] Server: Also generates a "Awesome!" chat message.

[0264] 5. Chat sending and display

[0265] Server: Sends the generated "Like" and "Awesome" chat messages to the device.

[0266] Terminal: Chat messages are displayed on the user's screen.

[0267] This system will increase interactivity between speakers and participants in online study groups and lectures, and provide emotion-based feedback, resulting in more meaningful and effective communication.

[0268] The processing flow will be explained below.

[0269] Program processing steps

[0270] Step 1:

[0271] The user accesses the online study group link and enters their login information.

[0272] The terminal sends the entered login information to the server.

[0273] The server checks the authentication information against a database to verify that the participant is legitimate.

[0274] Step 2:

[0275] The user (speaker) begins speaking using a device with a microphone.

[0276] The device collects the speaker's voice in real time and transmits it to the server as digital audio data.

[0277] Step 3:

[0278] The server sends the received voice data to the voice recognition engine.

[0279] A voice recognition engine converts the voice data into text data.

[0280] The server receives the converted text data.

[0281] Step 4:

[0282] The server passes the text data to a natural language processing (NLP) module.

[0283] The NLP module analyzes the context of the text data and determines appropriate backchannel phrases.

[0284] The server selects the appropriate response based on the analysis results.

[0285] Step 5:

[0286] The server sends the collected voice data and converted text data to the emotion engine.

[0287] The emotion engine analyzes the user's emotions and selects the most appropriate response phrase based on the results.

[0288] The server generates the selected backchannel phrases and constructs them as chat messages.

[0289] Step 6:

[0290] The server sends the generated chat message to the device.

[0291] The terminal displays the sent chat message on the user's screen.

[0292] Specific examples

[0293] Example 1: A speaker is explaining a new theory

[0294] Step 1:

[0295] The user (speaker) says, "Based on this theory, performance will improve significantly."

[0296] The terminal collects the speaker's voice and sends it to the server.

[0297] Step 2:

[0298] The server receives the voice data and sends it to the voice recognition engine.

[0299] The speech recognition engine converts the speech into text and generates text data that says, "Based on this theory, performance will be significantly improved."

[0300] The server receives the conversion result.

[0301] Step 3:

[0302] The server passes the text data to the NLP module.

[0303] The NLP module analyzes the context and determines the "like" response.

[0304] The server selects "Likes" based on the analysis results.

[0305] Step 4:

[0306] The server sends the text data and voice data to the emotion engine.

[0307] The emotion engine analyzes the user's emotions and detects positive emotions.

[0308] The emotion engine also determines that the response "Amazing!" is appropriate and selects it.

[0309] The server generates "Like" and "Awesome" chat messages.

[0310] Step 5:

[0311] The server sends the generated chat message to the device.

[0312] The terminal displays the chat messages on the user's screen.

[0313] As shown in this example, the system converts the speaker's speech into text and generates and displays appropriate responses using natural language processing and sentiment analysis. This process improves interaction in online study groups and lectures, promoting two-way communication between speakers and participants.

[0314] Example 2

[0315] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0316] Online study groups and lectures face a lack of interactive communication between speakers and participants. In particular, there is a lack of a way to automatically generate appropriate responses and provide emotion-based feedback to the speaker. This makes it difficult for participants to respond appropriately to the content of the lecture, and the lecture tends to become a one-way communication of information.

[0317] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0318] In this invention, the server includes means for collecting the speaker's voice, means for converting the collected voice into text data, means for analyzing the context of the text data and selecting appropriate backchannels, means for optimizing the selected backchannels based on emotion analysis, means for sending the optimized backchannels as chat messages, and means for displaying the chat messages on a screen. This enables interactive communication with participants in online study groups and lectures by automatically generating appropriate backchannels based on emotion.

[0319] "Audio collection means" refers to a device or technology that captures the speaker's voice as digital data and transmits that data to a server.

[0320] The "means for converting voice into text data" refers to a voice recognition technology or system that automatically converts collected voice data into text data.

[0321] "Means for analyzing context and selecting appropriate backchannels" refers to natural language processing (NLP) technology or systems that analyze the content of text data and determine appropriate backchannel phrases.

[0322] The "means for optimizing based on emotion analysis" is an emotion engine or algorithm that analyzes the user's emotions and selects and optimizes appropriate responses based on the results.

[0323] The "means for transmitting backchannels as chat messages" refers to a communication technology or system that transmits the generated backchannels to a terminal in the form of a chat message.

[0324] The "means for displaying chat messages on a screen" refers to a technology or system for displaying sent chat messages on the terminals of participants.

[0325] This invention is a system for promoting interactive communication between speakers and participants in online study groups and lectures. In particular, by combining an emotion engine to analyze the user's emotions and generate and display appropriate responses, more lively communication is possible.

[0326] Hardware and software used

[0327] 1. A device with a microphone (e.g., a headset or smartphone)

[0328] 2. Server (including speech recognition engine, natural language processing engine, emotion engine)

[0329] 3. Devices (PC, smartphone, tablet)

[0330] Detailed explanation of the process

[0331] First, the user (speaker) begins their lecture using a device with a microphone. The device collects the speaker's voice in real time and transmits it to the server as digital audio data. For example, the speaker's utterance, "Good morning, today I will talk about the future of AI," is collected by the microphone and transmitted as digital data to the server.

[0332] Next, the server sends the received voice data to a speech recognition engine and converts the voice data into text data. Specifically, it uses the Google Cloud Speech-to-Text API to convert the voice data into text data such as "Good morning, today I'll talk about the future of AI." The server receives this converted text data.

[0333] The server then passes this text data to a natural language processing (NLP) module for context analysis. Using Hugging Face's Transformers library, the server analyzes the context of the text data and determines appropriate backchannel phrases. For example, it selects "Interesting!" from the context "Today we'll talk about the future of AI."

[0334] The server then passes the collected voice and text data to the emotion engine. Using IBM Watson's (registered trademark) Tone Analyzer, the engine analyzes the user's emotions and selects the most appropriate backchannel response based on the results. For example, if the speaker is analyzed as speaking with confidence, the engine selects an additional positive backchannel response such as "That's great! I'd like to hear more!" By sending this backchannel response, it becomes possible to provide feedback based on the speaker's emotions.

[0335] In the final stage, the server sends the generated chat message to the device, which displays it on the user's screen so that all participants can see it. For example, chat messages such as "Interesting!" and "Awesome, I'd love to hear more!" are displayed and shared with participants in real time.

[0336] Specific examples

[0337] Example: A speaker is explaining a new theory.

[0338] 1. User (speaker): "Based on this theory, performance will improve significantly."

[0339] 2. Device: Collects audio from the microphone and sends it to the server.

[0340] 3. Server: Sends the audio data to the Google Cloud Speech-to-Text API and receives it as text data.

[0341] 4. Server: The text data is analyzed using Hugging Face's Transformers library to determine whether it is a "Like!"

[0342] 5. Server: Use IBM Watson's Tone Analyzer to analyze emotions and select additional "Amazing!" responses.

[0343] 6. Server: Sends "Like" and "Awesome" chat messages to the device.

[0344] 7. Terminal: Chat messages are displayed on the user's screen so that participants can see them.

[0345] Prompt Sentence Examples

[0346] The speaker will talk about "The Future of AI Technology." Part of the lecture will include the following:

[0347] Speaker remarks: "These technologies have the potential to profoundly change the way we live our lives."

[0348] System-based text conversion: "These technologies have the potential to significantly change our lives."

[0349] System analysis: "Great!"

[0350] Sentiment analysis by the system: "Amazing!"

[0351] Expected output:

[0352] When the speaker says, "These technologies have the potential to dramatically change our lives," the responses "Amazing!" and "Amazing!" are displayed.

[0353] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0354] Step 1:

[0355] Collecting speaker audio

[0356] User (speaker): Starts speaking using a device with a microphone.

[0357] Input: Speaker audio from a device with a microphone.

[0358] Output: Audio data in digital format.

[0359] Specific behavior:

[0360] The speaker says, "Good morning, today I'll be talking about the future of AI." The device (computer or smartphone app) captures the voice with its built-in microphone, converts the voice data into digital format, and transmits it to a server in real time.

[0361] Step 2:

[0362] Converting audio data to text

[0363] Server: Sends the received voice data to the speech recognition engine.

[0364] Input: Audio data in digital format.

[0365] Output: Text data.

[0366] Specific behavior:

[0367] The server sends the received voice data to the Google Cloud Speech-to-Text API and converts it into text data such as "Good morning, today I'll talk about the future of AI." The converted text data is then received by the server.

[0368] Step 3:

[0369] Context analysis and backchannel generation

[0370] Server: Passes the text data to a natural language processing (NLP) module for contextual analysis.

[0371] Input: The converted text data.

[0372] Output: An appropriate backchannel phrase.

[0373] Specific behavior:

[0374] The server passes the text data to an NLP module using Hugging Face's Transformers library. The NLP module analyzes the context, "Today we'll talk about the future of AI," and selects an appropriate back-channel phrase, such as "Interesting!" The server then generates a chat message saying, "Interesting!"

[0375] Step 4:

[0376] Sentiment analysis and backchannel optimization

[0377] Server: Passes the collected voice and text data to the emotion engine.

[0378] Input: Audio and text data.

[0379] Output: Optimized backchannel phrases.

[0380] Specific behavior:

[0381] The server passes the audio and text data to IBM Watson's Tone Analyzer. If the emotion engine determines that the speaker is speaking confidently, it selects a positive response such as, "Great, I'd love to hear more!" The server then generates this new response as a chat message.

[0382] Step 5:

[0383] Sending and viewing chat messages

[0384] Server: Sends generated chat messages to devices.

[0385] Input: Optimized backchannel phrase.

[0386] Output: The chat messages that are displayed on the user's screen.

[0387] Specific behavior:

[0388] The server sends the generated chat messages, such as "Interesting!" and "Amazing, I'd love to hear more!", to the device. The device (participant's PC or smartphone) displays the chat messages on the user's screen. All participants can view the messages in real time through the chat window.

[0389] (Application example 2)

[0390] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0391] In online study groups and lectures, there is a lack of interactive communication between speakers and participants, so more active and meaningful communication is required. In addition, there is a challenge in providing appropriate feedback on audience emotions and reactions to improve the quality of real-time interactions.

[0392] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting the speaker's voice, means for converting the collected voice into text data, means for analyzing the context of the text data and selecting appropriate backchannels, means for sending the selected backchannels as chat messages, means for displaying the chat messages on a screen, means for collecting viewer comments and analyzing the context and emotions of the comments, and means for generating and displaying appropriate backchannels and reactions to the viewer based on the analysis results. This allows for real-time exchange of emotion-based feedback between the speaker, participants, viewers, etc., enabling more fulfilling interactive communication.

[0393] A "speaker" is a person who speaks at an online study group or lecture.

[0394] "Means for collecting audio" refers to devices and methods for recording the speaker's speech and transmitting it as digital data to a server.

[0395] "Means for converting voice into text data" refers to the technology or method for analyzing collected voice data and converting it into text format.

[0396] "Means for analyzing the context of text data" refers to natural language processing techniques and methods for analyzing text data and understanding its content and meaning.

[0397] "Means for selecting backchannels" refers to techniques and methods for selecting appropriate backchannels and responses based on the results of context analysis and emotion analysis.

[0398] "Means for sending as a chat message" refers to a technique or method for sending the generated backchannels or responses as an electronic message.

[0399] The "means for displaying chat messages on a screen" refers to a technique or method for displaying the sent messages on a display device.

[0400] "Means for collecting viewer comments" refers to techniques and methods for collecting comments from viewers in real time.

[0401] "Means for analyzing the context and sentiment of comments" refers to natural language processing and sentiment analysis technologies that analyze comments from viewers and understand their content and emotional state.

[0402] "Means for generating appropriate responses and reactions to viewers based on the analysis results" refers to technology or methods that automatically generate appropriate responses and reactions to viewers' comments based on the analysis results.

[0403] This invention provides a system for promoting interactive communication between a lecturer and an audience in an online study group or lecture meeting. The operation of this system will be described in detail below.

[0404] 1. Audio collection and conversion

[0405] A device with a microphone is used to collect the speaker's voice. The user (speaker) starts their lecture or explanation, and the device collects the speaker's voice in real time and sends this audio data in digital format to a server. The server then converts the received audio data into text data using a speech recognition engine (e.g., Google Speech-to-Text API).

[0406] 2. Context Analysis and Backchannel Generation

[0407] The server passes the text data to a natural language processing (NLP) module (such as SpaCy or NLTK) to analyze the context. The NLP module analyzes the context of the text data and determines appropriate backchannel phrases (e.g., "Uh-huh," "I see," "Great!", "Amazing!"). Based on the analysis results, the server generates appropriate backchannels.

[0408] 3. Collection and analysis of viewer comments

[0409] The server uses real-time communication technology (e.g., WebSocket) to collect comments from viewers in real time. After collecting the comments, these comments are also passed to the NLP module to analyze the context and sentiment. For sentiment analysis, an emotion engine (e.g., IBM Watson Tone Analyzer) is used.

[0410] 4. Reaction generation and display

[0411] The server generates appropriate responses and reactions for the audience based on the analysis results. These responses and reactions are sent as chat messages, which are then displayed on the screens of the user (speaker) and audience members. This allows the speaker to see audience reactions in real time, and audience members can receive feedback on their own comments.

[0412] Specific examples

[0413] Below is a specific example scenario of a live stream where a speaker is talking about protecting the environment.

[0414] 1. Speaker: "We are being asked to reduce our use of plastic by 50% over the next 10 years."

[0415] 2. Real-time analysis and backchannel generation:

[0416] The speech recognition engine converts speech into text.

[0417] The NLP module analyzes the context and generates an appropriate response: "That's important!"

[0418] The sentiment analysis engine analyzes and emphasizes the speaker's passion, adding "What an amazing initiative!"

[0419] 3. Feedback display:

[0420] The speaker sees these responses in real time and continues speaking with even more enthusiasm.

[0421] Prompt Sentence Examples

[0422] Below is an example of a prompt sentence that is input to a generative AI model.

[0423] Speaker's audio data

[0424] There is a call to reduce plastic use by 50% over the next 10 years.

[0425] prompt

[0426] Generate appropriate responses to what this speaker has said based on audience sentiment analysis.

[0427] Generated responses (example)

[0428] That's important!

[0429] What an amazing initiative!

[0430] This allows for real-time emotional feedback between the speaker and the audience, enabling richer, more interactive communication.

[0431] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0432] Step 1:

[0433] The speaker (user) starts a lecture or explanation using a device with a microphone. The device collects the speaker's voice in real time, converts this voice data into a digital format, and sends it to the server.

[0434] Input: Speaker's audio data

[0435] Output: Digital audio data

[0436] How it works: The device collects audio through a microphone, converts it into digital form, and sends it over the network to a server.

[0437] Step 2:

[0438] The server passes the received voice data to a voice recognition engine (for example, Google Speech-to-Text API) and converts the voice data into text data.

[0439] Input: Digital audio data

[0440] Output: Text data

[0441] Specific operation: The server sends audio data to the Google Speech-to-Text API and receives the converted text data.

[0442] Step 3:

[0443] The server passes the text data to a natural language processing (NLP) module (e.g., SpaCy or NLTK) for context analysis, which then determines the appropriate backchannel phrases.

[0444] Input: Text data

[0445] Output: Context analysis results and backchannel phrases

[0446] What it does: The server passes the text data to the NLP module, which generates appropriate back-channel responses (e.g., "Uh-huh," "I see," "Great!").

[0447] Step 4:

[0448] The server formats the generated backchannel phrases as chat messages and sends them to the terminal.

[0449] Input: Backchannel phrase

[0450] Output: Chat message

[0451] Specific operation: The server converts the generated backchannel phrases into chat messages and sends them to each viewer's device.

[0452] Step 5:

[0453] The terminal displays the received chat messages on the screen so that the speaker and the audience can see the messages.

[0454] Input: Chat message

[0455] Output: Chat messages displayed

[0456] Specific operation: The device displays the received chat message on the user's screen.

[0457] Step 6:

[0458] The server collects comments from viewers in real time and passes these comments to a natural language processing (NLP) module for context and sentiment analysis.

[0459] Input: Viewer comments

[0460] Output: Contextual and sentiment analysis results

[0461] Specific operation: The server collects comments using real-time communication technology (such as WebSocket) and passes them to the NLP module and emotion engine for analysis.

[0462] Step 7:

[0463] The server generates appropriate responses and reactions based on the viewers' comments, which are then formatted as chat messages and sent to the device.

[0464] Input: Contextual and sentiment analysis results

[0465] Output: Chat message

[0466] Specific operation: The server generates responses and reactions to the comments based on the analysis results, formats them as chat messages, and sends them.

[0467] Step 8:

[0468] The device displays the generated responses and reactions on the viewer's screen, allowing them to see the feedback in real time.

[0469] Input: Chat message

[0470] Output: The displayed responses and reactions

[0471] What it does: The device displays the received responses and reactions on the viewer's screen, providing real-time feedback.

[0472] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0473] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0474] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0475] [Second embodiment]

[0476] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0477] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0478] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0479] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0480] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0481] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0482] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0483] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0484] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0485] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0486] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0487] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0488] This invention is a system for realizing interactive communication between speakers and participants in online study groups and lectures. Specifically, it collects the speaker's voice, converts that voice into text, analyzes the context of the text, and automatically generates and transmits appropriate responses.

[0489] This system mainly consists of the following devices and technologies:

[0490] 1. Collecting speaker audio

[0491] The user (speaker) uses a device with a microphone (computer, smartphone, tablet, etc.) to give a lecture or explanation.

[0492] The device collects the speaker's voice in real time and transmits it to the server as digital audio data.

[0493] 2. Converting voice data to text

[0494] The server converts the received voice data into text data using speech recognition technology. This process uses a speech recognition engine (such as the Google Speech-to-Text API).

[0495] The server receives the converted text data and prepares it for contextual analysis.

[0496] 3. Context Analysis and Backchannel Generation

[0497] The server uses natural language processing technology to analyze the context of the text data, identifying what the speaker said and when, and then selects appropriate back-channel responses (e.g., "Uh-huh," "I see," "Great!", "Amazing!", etc.).

[0498] The server automatically generates the selected responses in the form of chat messages.

[0499] 4. Sending and Viewing Chat Messages

[0500] The server transmits the generated backchannel message to the terminal.

[0501] The terminal displays the sent chat message on the user's screen so that all participants can see the message.

[0502] Specific examples

[0503] Example 1: A speaker is explaining a theory

[0504] 1. Audio collection

[0505] User: "Based on this theory, performance will improve significantly."

[0506] Device: Collects audio from the microphone and sends it to the server.

[0507] 2. Text Conversion

[0508] Server: Sends the speech data to the speech recognition engine and receives the text data, "Based on this theory, performance will be significantly improved."

[0509] 3. Context Analysis and Backchannel Generation

[0510] Server: Analyzes the text data using natural language processing technology and selects the appropriate "Like!" response phrase based on the context.

[0511] Server: Generate a "Like" chat message.

[0512] 4. Chat sending and display

[0513] Server: Sends the generated "like" chat message to the device.

[0514] Terminal: Chat messages are displayed on the user's screen.

[0515] In this way, appropriate responses are automatically inserted while the speaker is speaking, making it easier for participants to actively engage with the content of the lecture, and the speaker can also see participants' reactions in real time. This system improves the quality of online communication and provides a more effective learning environment.

[0516] The processing flow will be explained below.

[0517] Step 1:

[0518] The user accesses the online study group link and enters their login information.

[0519] The terminal sends the entered login information to the server.

[0520] The server checks the authentication information against a database to verify that the participant is legitimate.

[0521] Step 2:

[0522] The user (speaker) uses a device with a microphone to begin a lecture or explanation.

[0523] The device collects the speaker's audio in real time.

[0524] The device transmits the collected voice data to the server in real time.

[0525] Step 3:

[0526] The server sends the received voice data to the voice recognition engine.

[0527] A voice recognition engine converts the voice data into text data.

[0528] The server receives the converted text data.

[0529] Step 4:

[0530] The server passes the text data to a natural language processing (NLP) module.

[0531] The NLP module analyzes the context of the text data and determines appropriate responses.

[0532] The server selects appropriate response phrases based on the analysis results.

[0533] Step 5:

[0534] The server generates the selected backchannel as a chat message.

[0535] The server sends the generated chat message to the device.

[0536] The terminal displays the chat messages on the user's screen.

[0537] Step 6:

[0538] Users (participants) check the chat messages and enter their reactions or questions in the chat field as necessary.

[0539] The device sends the input responses and questions to the server.

[0540] The server collects participants' responses and questions and uses them to generate subsequent responses and analyze context.

[0541] Example 1

[0542] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0543] In online study groups and lectures, there is a tendency for there to be a lack of interaction between the speaker and the participants, which can lead to a decline in the quality of communication. This makes it difficult for the speaker to get participants' reactions in real time, and for participants to maintain interest in the content of the lecture.

[0544] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0545] In this invention, the server includes means for collecting the speaker's voice, means for converting the collected voice into text data, means for analyzing the context of the text data and selecting appropriate backchannels, means for transmitting the selected backchannels as chat messages, and means for displaying the chat messages on a screen. This allows the speaker to receive appropriate backchannels from the participants in real time, and allows the participants to maintain their interest in the content of the lecture.

[0546] "Means for collecting the speaker's voice" refers to a device or function that uses the microphone of the device (computer, smartphone, tablet, etc.) used by the speaker to collect the voice during the lecture as digital voice data in real time.

[0547] "Means for converting collected voice data into text data" refers to technology that uses a voice recognition engine to convert collected digital voice data into text data. For example, Google Speech-to-Text API is one such technology.

[0548] "Means for analyzing the context of text data and selecting appropriate backchannels" refers to technologies and algorithms for analyzing the content and context of converted text data using natural language processing technology and selecting appropriate backchannel phrases.

[0549] "Means for transmitting the selected backchannels as chat messages" refers to technologies and protocols for converting the selected backchannel phrases into chat message format and transmitting them to participants' devices. Examples include WebSocket and HTTP.

[0550] "Means for displaying chat messages on a screen" refers to software and technology for displaying chat messages sent to participants' devices on an interface in real time. For example, this includes online conference tools and dedicated applications.

[0551] "Speech recognition technology" refers to technology for converting voice data into text data, including voice models and machine learning algorithms. An example is the Google Speech-to-Text API.

[0552] "Natural language processing technology" is a technology for analyzing the content and context of text data and understanding its meaning, and includes technologies such as morphological analysis, semantic analysis, and sentiment analysis. Models such as BERT are also part of this technology.

[0553] This invention is a system for realizing interactive communication between speakers and participants in online study groups and lectures. This system collects the speaker's voice, converts the voice into text, analyzes the context of the text, and automatically generates and transmits appropriate responses. The specific configuration and implementation method of this system are described in detail below.

[0554] Hardware and Software Use

[0555] 1. Audio collection

[0556] The user (speaker) gives a lecture or explanation using a device with a microphone (computer, smartphone, tablet, etc.).

[0557] The device uses a microphone to collect the speaker's voice in real time and uses an internet connection to send it to a server as digital audio data.

[0558] 2. Text Conversion

[0559] The server converts the received voice data into text data using speech recognition technology, using a speech recognition engine such as the Google Speech-to-Text API.

[0560] The server receives the converted text data and prepares it for contextual analysis.

[0561] 3. Context Analysis and Backchannel Generation

[0562] The server analyzes the context of the text data using natural language processing (NLP) techniques, including morphological analysis, semantic analysis, and sentiment analysis, and also uses AI models such as BERT.

[0563] The server selects appropriate backchannel phrases based on the analysis results. The selected backchannels are either chosen from a predefined list of phrases or dynamically generated by AI.

[0564] 4. Sending and Viewing Chat Messages

[0565] The server sends the generated backchannel message to the device using a communication protocol such as WebSocket or HTTP.

[0566] The device displays the sent chat messages on the user's screen using an online conference tool or a dedicated application.

[0567] Specific examples

[0568] For example, here is the flow when a speaker explains, "Based on this theory, performance will improve significantly."

[0569] 1. Audio collection

[0570] User: "Based on this theory, performance will improve significantly."

[0571] Device: Collects audio from the microphone and sends it to the server.

[0572] 2. Text Conversion

[0573] Server: Sends audio data to the Google Speech-to-Text API and receives text data that says, "Based on this theory, performance will be significantly improved."

[0574] 3. Context Analysis and Backchannel Generation

[0575] Server: Analyzes the text data using natural language processing technology and selects the appropriate "Like!" response phrase based on the context.

[0576] Server: Generate a "Like" chat message.

[0577] 4. Sending and Viewing Chat Messages

[0578] Server: Sends the generated "like" chat message to the device.

[0579] Terminal: A web application displays chat messages on the user's screen.

[0580] This system allows users to receive appropriate responses from participants in real time during a lecture, and also helps participants to become more interested in the content of the lecture, thereby improving the quality of online communication and providing a more effective learning environment.

[0581] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0582] Step 1: Audio Collection

[0583] Input: A user speaks using the microphone on their device (computer, smartphone, tablet, etc.).

[0584] What happens: The speaker gives a presentation or talk, and the microphone collects the audio.

[0585] Device operation: The device's microphone collects the speaker's voice in real time.

[0586] Output: Collected digital audio data is generated and used in the next processing step.

[0587] Step 2: Sending audio data

[0588] Input: Digital audio data collected by the device.

[0589] What happens next: Your device sends audio data to a server over your internet connection.

[0590] Device operation: The device prepares the collected digital audio data and initiates communication to send it to the server.

[0591] Output: Audio data is sent to and received by the server.

[0592] Step 3: Convert audio data to text

[0593] Input: Digital audio data received by the server.

[0594] Specific actions taken: The server uses speech recognition technology (e.g., Google Speech-to-Text API) to convert the voice data into text data.

[0595] Server operation: The server sends the voice data to the speech recognition engine, which analyzes the voice and generates text data.

[0596] Output: The audio data is converted into "text data" and used in the next context analysis step.

[0597] Step 4: Contextual analysis of text data

[0598] Input: Text data returned from the speech recognition engine.

[0599] Specific actions taken: The server uses natural language processing techniques to analyze the context of the text data, including morphological analysis, semantic analysis, and sentiment analysis.

[0600] Server operation: The server analyzes the text data, performs calculations to identify the content and timing of the speaker's remarks, and extracts key keywords and sentiment.

[0601] Output: Information based on the contextual analysis is generated and used in the next step of backchannel generation.

[0602] Step 5: Selecting and generating backchannel phrases

[0603] Input: Information based on contextual analysis.

[0604] What happens: The server selects appropriate backchannels based on the contextual analysis results. The selected backchannels are either chosen from a pre-defined list or dynamically generated by a generative AI model.

[0605] Server operation: The server selects appropriate backchannel phrases based on the analysis results and automatically generates them in the form of chat messages.

[0606] Output: The selected backchannel phrase is generated as a chat message and used in the next sending step.

[0607] Step 6: Send a chat message

[0608] Input: An automatically generated chat backchannel message.

[0609] Specific actions taken: The server sends chat messages over the Internet to the user's device.

[0610] Server operation: The server initiates communication to send the generated chat message to the specified device. WebSocket or HTTP is used as the communication protocol.

[0611] Output: The chat message is sent to the device.

[0612] Step 7: View chat messages

[0613] Input: The chat message sent by the server.

[0614] Specific actions taken: Display chat messages received by the device on the screen.

[0615] Terminal Action: Displays a new message in a chat message window or similar, allowing all participants to see the message in real time.

[0616] Output: A backchannel message will be displayed in the chat window.

[0617] This series of processes allows users to receive appropriate responses from participants in real time during a lecture, making it easier for participants to become interested in the content of the lecture. This improves the quality of online communication and provides a more effective learning environment.

[0618] (Application example 1)

[0619] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0620] In online shopping virtual stores, customers often have difficulty getting a proper response in real time when they have questions or comments about products, which can lead to a poor customer experience and reduced purchasing intent.

[0621] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0622] In this invention, the server includes means for collecting the speaker's voice, means for converting the collected voice into text data, means for analyzing the context of the text data and selecting appropriate backchannels, means for sending the selected backchannels as chat messages, means for displaying the chat messages on a screen, means for collecting customer voice, converting it into text, analyzing the context and generating a response, and means for displaying the generated response message on a display device, thereby improving interaction with customers in real time and enabling a better shopping experience.

[0623] The "means for collecting the speaker's voice" refers to a device or method for collecting the speaker's voice in real time using a microphone device or the like and converting it into digital voice data.

[0624] The "means for converting collected voice data into text data" refers to a device or method for converting collected voice data into text information using voice recognition technology.

[0625] "Means for analyzing the context of text data and selecting appropriate responses" refers to a device or method that uses natural language processing technology to analyze the content and timing of statements contained in text data, and generates or selects appropriate responses or comments based on that context.

[0626] The "means for transmitting the selected backchannel as a chat message" refers to a device or method for transmitting the selected backchannel or response message in a chat format via a network.

[0627] The "means for displaying chat messages on a screen" refers to a device or method for displaying received chat messages on a user interface so that the user can check their contents.

[0628] "Means for collecting customer speech, converting it into text, analyzing the context, and generating a response" refers to a device or method that collects speech uttered by a customer, converts it into text information, analyzes the context, and automatically generates an appropriate response.

[0629] "Means for displaying the generated response message on a display device" refers to a device or method that outputs the generated response message to a display or other display device so that the user can visually confirm the message.

[0630] This invention is a system that automatically supports interactive communication in online lectures and virtual stores using speech recognition and context analysis. This system collects user speech, converts it into text in real time, analyzes the context, generates appropriate responses, and displays them on a display device. To function effectively, this system requires the following hardware and software:

[0631] Hardware and Software

[0632] Microphone devices: Used to collect audio data. This includes computers, smartphones, tablets, etc. with microphones.

[0633] Computer server: Receives the voice data and analyzes it using speech recognition and natural language processing technologies. It converts the voice data into text using Google Cloud's Speech-to-Text API and applies the GPT-3.5-turbo model using the transformers library.

[0634] Display device: A display device that shows chat messages and reply messages to users.

[0635] Data processing flow

[0636] 1. Audio collection: The device (user's smartphone or computer) collects the user's audio through a microphone device and transmits it to the server as digital audio data.

[0637] 2. Converting speech data to text: The server converts the speech data received into text using Google Cloud's Speech-to-Text API and prepares it for contextual analysis.

[0638] 3. Contextual analysis and response generation: The server uses natural language processing techniques to analyze the context of the text data, and uses the GPT-3.5-turbo model to generate appropriate responses based on that analysis.

[0639] 4. Creating and displaying a chat message or response message: The server sends the generated response to the terminal, which displays the message on its display device.

[0640] Specific examples

[0641] Example 1: Virtual store example

[0642] 1. Audio collection:

[0643] Customer: "Do you have this jacket in any other colors?"

[0644] Device: Collects audio from the microphone and sends it to the server.

[0645] 2. Text conversion:

[0646] Server: Sends the voice data to the voice recognition engine and receives the text data "Do you have this jacket in other colors?"

[0647] 3. Context analysis and response generation:

[0648] Server: Analyzes the text data using natural language processing technology and generates a response based on the context: "Of course we have it. Currently, it comes in three colors: black, blue, and red."

[0649] 4. Chat sending and display:

[0650] Server: Sends the generated response message to the terminal.

[0651] Terminal: Displays the response message on the user's screen.

[0652] Example prompt sentence:

[0653] Customer: Do you have this jacket in other colors?

[0654] context:

[0655] Generate a response.

[0656] The present invention improves customer experience in virtual stores and enables real-time interactive communication.

[0657] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0658] Step 1:

[0659] Audio collection: The device (user's smartphone or computer) collects the user's voice using a microphone device. When the user speaks, the microphone captures it and converts it into digital audio data in real time. This digital audio data is sent to the next processing step.

[0660] Input: User's voice

[0661] Output: Digital audio data

[0662] What it does: The microphone captures audio and the application converts it into a digital format.

[0663] Step 2:

[0664] Speech-to-text conversion: The device sends the collected digital speech data to a server, which converts the speech to text using Google Cloud's Speech-to-Text API and prepares the converted text for analysis.

[0665] Input: Digital audio data

[0666] Output: Text data

[0667] What happens: The server sends the audio data to the Google Cloud Speech-to-Text API, which converts the audio into text.

[0668] Step 3:

[0669] Contextual analysis and response generation: The server receives the text data and analyzes its context using natural language processing techniques. This analysis uses the transformers library and the GPT-3.5-turbo model. The generative AI model generates an appropriate response prompt from the analysis results.

[0670] Input: Text data

[0671] Output: Response message

[0672] Specific operation: The server sends a prompt to the GPT-3.5-turbo model using text data as input, and the model generates a response message.

[0673] Step 4:

[0674] Sending a response message: The server sends the generated response message to the terminal, which receives it and prepares it for display on the display device.

[0675] Input: Response message

[0676] Output: Response message sent to the terminal

[0677] Specific operation: The server sends a response message to the terminal over the network.

[0678] Step 5:

[0679] Display: The terminal displays the received response message on a display device, allowing the user to see the generated response in real time.

[0680] Input: Response message sent to the terminal

[0681] Output: Response message displayed on display device

[0682] Specific actions: The terminal renders the response message on the screen and displays it on the display device.

[0683] This enables real-time two-way communication between users and the system, improving the customer experience in the virtual store.

[0684] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0685] This invention is a system for promoting interactive communication between speakers and participants in online study groups and lectures. In particular, it combines an emotion engine to analyze the user's emotions and generate and display appropriate responses, enabling more lively communication.

[0686] This system is composed of the following major devices and technologies:

[0687] 1. Collecting speaker audio

[0688] The user (speaker) begins a lecture or explanation using a device with a microphone.

[0689] The terminal collects the speaker's voice in real time and transmits it to the server as digital voice data.

[0690] 2. Converting voice data to text

[0691] The server transmits the received voice data to a voice recognition engine, which converts the voice data into text data.

[0692] The speech recognition engine returns the converted text data to the server, which receives it.

[0693] 3. Context Analysis and Backchannel Generation

[0694] The server passes the text data to a natural language processing (NLP) module for contextual analysis.

[0695] The NLP module analyzes the context of the text data and determines appropriate interjections (e.g., "Uh-huh," "I see," "Great!", "Amazing!", etc.).

[0696] The server selects and generates appropriate responses based on the analysis results.

[0697] 4. Sentiment analysis and backchannel optimization

[0698] The server passes the collected voice data and text data to the emotion engine.

[0699] The emotion engine analyzes emotions from the user's voice and text, and selects appropriate responses based on the results.

[0700] For example, through emotion analysis, if a speaker is speaking enthusiastically, the system will generate many positive responses such as "Amazing!", but if the listeners are feeling stressed, it will generate neutral or empathetic responses such as "I see" or "Did you understand?"

[0701] 5. Sending and Viewing Chat Messages

[0702] The server sends the generated chat message to the device.

[0703] The terminal displays the sent chat message on the user's screen so that all participants can see the message.

[0704] Specific examples

[0705] Example 1: A speaker is explaining a new theory

[0706] 1. Audio collection

[0707] User (speaker): "Based on this theory, performance will improve significantly."

[0708] Device: Collects audio from the microphone and sends it to the server.

[0709] 2. Text Conversion

[0710] Server: Sends the speech data to the speech recognition engine and receives the text data, "Based on this theory, performance will be significantly improved."

[0711] 3. Context Analysis and Backchannel Generation

[0712] Server: Analyzes the text data using natural language processing technology and selects a "Like!" response based on the context.

[0713] Server: Generate a "Like" chat message.

[0714] 4. Sentiment analysis and backchannel optimization

[0715] Server: Passes voice and text data to the emotion engine.

[0716] Emotion Engine: If a speaker is speaking confidently, it analyzes positive emotions and selects additional back-channel responses like "Awesome!" to emphasize them.

[0717] Server: Also generates a "Awesome!" chat message.

[0718] 5. Chat sending and display

[0719] Server: Sends the generated "Like" and "Awesome" chat messages to the device.

[0720] Terminal: Chat messages are displayed on the user's screen.

[0721] This system will increase interactivity between speakers and participants in online study groups and lectures, and provide emotion-based feedback, resulting in more meaningful and effective communication.

[0722] The processing flow will be explained below.

[0723] Program processing steps

[0724] Step 1:

[0725] The user accesses the online study group link and enters their login information.

[0726] The terminal sends the entered login information to the server.

[0727] The server checks the authentication information against a database to verify that the participant is legitimate.

[0728] Step 2:

[0729] The user (speaker) begins speaking using a device with a microphone.

[0730] The device collects the speaker's voice in real time and transmits it to the server as digital audio data.

[0731] Step 3:

[0732] The server sends the received voice data to the voice recognition engine.

[0733] A voice recognition engine converts the voice data into text data.

[0734] The server receives the converted text data.

[0735] Step 4:

[0736] The server passes the text data to a natural language processing (NLP) module.

[0737] The NLP module analyzes the context of the text data and determines appropriate backchannel phrases.

[0738] The server selects the appropriate response based on the analysis results.

[0739] Step 5:

[0740] The server sends the collected voice data and converted text data to the emotion engine.

[0741] The emotion engine analyzes the user's emotions and selects the most appropriate response phrase based on the results.

[0742] The server generates the selected backchannel phrases and constructs them as chat messages.

[0743] Step 6:

[0744] The server sends the generated chat message to the device.

[0745] The terminal displays the sent chat message on the user's screen.

[0746] Specific examples

[0747] Example 1: A speaker is explaining a new theory

[0748] Step 1:

[0749] The user (speaker) says, "Based on this theory, performance will improve significantly."

[0750] The terminal collects the speaker's voice and sends it to the server.

[0751] Step 2:

[0752] The server receives the voice data and sends it to the voice recognition engine.

[0753] The speech recognition engine converts the speech into text and generates text data that says, "Based on this theory, performance will be significantly improved."

[0754] The server receives the conversion result.

[0755] Step 3:

[0756] The server passes the text data to the NLP module.

[0757] The NLP module analyzes the context and determines the "like" response.

[0758] The server selects "Likes" based on the analysis results.

[0759] Step 4:

[0760] The server sends the text data and voice data to the emotion engine.

[0761] The emotion engine analyzes the user's emotions and detects positive emotions.

[0762] The emotion engine also determines that the response "Amazing!" is appropriate and selects it.

[0763] The server generates "Like" and "Awesome" chat messages.

[0764] Step 5:

[0765] The server sends the generated chat message to the device.

[0766] The terminal displays the chat messages on the user's screen.

[0767] As shown in this example, the system converts the speaker's speech into text and generates and displays appropriate responses using natural language processing and sentiment analysis. This process improves interaction in online study groups and lectures, promoting two-way communication between speakers and participants.

[0768] Example 2

[0769] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0770] Online study groups and lectures face a lack of interactive communication between speakers and participants. In particular, there is a lack of a way to automatically generate appropriate responses and provide emotion-based feedback to the speaker. This makes it difficult for participants to respond appropriately to the content of the lecture, and the lecture tends to become a one-way communication of information.

[0771] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0772] In this invention, the server includes means for collecting the speaker's voice, means for converting the collected voice into text data, means for analyzing the context of the text data and selecting appropriate backchannels, means for optimizing the selected backchannels based on emotion analysis, means for sending the optimized backchannels as chat messages, and means for displaying the chat messages on a screen. This enables interactive communication with participants in online study groups and lectures by automatically generating appropriate backchannels based on emotion.

[0773] "Audio collection means" refers to a device or technology that captures the speaker's voice as digital data and transmits that data to a server.

[0774] The "means for converting voice into text data" refers to a voice recognition technology or system that automatically converts collected voice data into text data.

[0775] "Means for analyzing context and selecting appropriate backchannels" refers to natural language processing (NLP) technology or systems that analyze the content of text data and determine appropriate backchannel phrases.

[0776] The "means for optimizing based on emotion analysis" is an emotion engine or algorithm that analyzes the user's emotions and selects and optimizes appropriate responses based on the results.

[0777] The "means for transmitting backchannels as chat messages" refers to a communication technology or system that transmits the generated backchannels to a terminal in the form of a chat message.

[0778] The "means for displaying chat messages on a screen" refers to a technology or system for displaying sent chat messages on the terminals of participants.

[0779] This invention is a system for promoting interactive communication between speakers and participants in online study groups and lectures. In particular, by combining an emotion engine to analyze the user's emotions and generate and display appropriate responses, more lively communication is possible.

[0780] Hardware and software used

[0781] 1. A device with a microphone (e.g., a headset or smartphone)

[0782] 2. Server (including speech recognition engine, natural language processing engine, emotion engine)

[0783] 3. Devices (PC, smartphone, tablet)

[0784] Detailed explanation of the process

[0785] First, the user (speaker) begins their lecture using a device with a microphone. The device collects the speaker's voice in real time and transmits it to the server as digital audio data. For example, the speaker's utterance, "Good morning, today I will talk about the future of AI," is collected by the microphone and transmitted as digital data to the server.

[0786] Next, the server sends the received voice data to a speech recognition engine and converts the voice data into text data. Specifically, it uses the Google Cloud Speech-to-Text API to convert the voice data into text data such as "Good morning, today I'll talk about the future of AI." The server receives this converted text data.

[0787] The server then passes this text data to a natural language processing (NLP) module for context analysis. Using Hugging Face's Transformers library, the server analyzes the context of the text data and determines appropriate backchannel phrases. For example, it selects "Interesting!" from the context "Today we'll talk about the future of AI."

[0788] The server then passes the collected voice and text data to the emotion engine. Using IBM Watson's Tone Analyzer, the engine analyzes the user's emotions and selects the most appropriate backchannel based on the results. For example, if the speaker is analyzed as speaking with confidence, the engine selects an additional positive backchannel such as "That's great! I'd like to hear more!" By sending this backchannel, it becomes possible to provide feedback based on the speaker's emotions.

[0789] In the final stage, the server sends the generated chat message to the device, which displays it on the user's screen so that all participants can see it. For example, chat messages such as "Interesting!" and "Awesome, I'd love to hear more!" are displayed and shared with participants in real time.

[0790] Specific examples

[0791] Example: A speaker is explaining a new theory.

[0792] 1. User (speaker): "Based on this theory, performance will improve significantly."

[0793] 2. Device: Collects audio from the microphone and sends it to the server.

[0794] 3. Server: Sends the audio data to the Google Cloud Speech-to-Text API and receives it as text data.

[0795] 4. Server: The text data is analyzed using Hugging Face's Transformers library to determine whether it is a "Like!"

[0796] 5. Server: Use IBM Watson's Tone Analyzer to analyze emotions and select additional "Amazing!" responses.

[0797] 6. Server: Sends "Like" and "Awesome" chat messages to the device.

[0798] 7. Terminal: Chat messages are displayed on the user's screen so that participants can see them.

[0799] Prompt Sentence Examples

[0800] The speaker will talk about "The Future of AI Technology." Part of the lecture will include the following:

[0801] Speaker remarks: "These technologies have the potential to profoundly change the way we live our lives."

[0802] System-based text conversion: "These technologies have the potential to significantly change our lives."

[0803] System analysis: "Great!"

[0804] Sentiment analysis by the system: "Amazing!"

[0805] Expected output:

[0806] When the speaker says, "These technologies have the potential to dramatically change our lives," the responses "Amazing!" and "Amazing!" are displayed.

[0807] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0808] Step 1:

[0809] Collecting speaker audio

[0810] User (speaker): Starts speaking using a device with a microphone.

[0811] Input: Speaker audio from a device with a microphone.

[0812] Output: Audio data in digital format.

[0813] Specific behavior:

[0814] The speaker says, "Good morning, today I'll be talking about the future of AI." The device (computer or smartphone app) captures the voice with its built-in microphone, converts the voice data into digital format, and transmits it to a server in real time.

[0815] Step 2:

[0816] Converting audio data to text

[0817] Server: Sends the received voice data to the speech recognition engine.

[0818] Input: Audio data in digital format.

[0819] Output: Text data.

[0820] Specific behavior:

[0821] The server sends the received voice data to the Google Cloud Speech-to-Text API and converts it into text data such as "Good morning, today I'll talk about the future of AI." The converted text data is then received by the server.

[0822] Step 3:

[0823] Context analysis and backchannel generation

[0824] Server: Passes the text data to a natural language processing (NLP) module for contextual analysis.

[0825] Input: The converted text data.

[0826] Output: An appropriate backchannel phrase.

[0827] Specific behavior:

[0828] The server passes the text data to an NLP module using Hugging Face's Transformers library. The NLP module analyzes the context, "Today we'll talk about the future of AI," and selects an appropriate back-channel phrase, such as "Interesting!" The server then generates a chat message saying, "Interesting!"

[0829] Step 4:

[0830] Sentiment analysis and backchannel optimization

[0831] Server: Passes the collected voice and text data to the emotion engine.

[0832] Input: Audio and text data.

[0833] Output: Optimized backchannel phrases.

[0834] Specific behavior:

[0835] The server passes the audio and text data to IBM Watson's Tone Analyzer. If the emotion engine determines that the speaker is speaking confidently, it selects a positive response such as, "Great, I'd love to hear more!" The server then generates this new response as a chat message.

[0836] Step 5:

[0837] Sending and viewing chat messages

[0838] Server: Sends generated chat messages to devices.

[0839] Input: Optimized backchannel phrase.

[0840] Output: The chat messages that are displayed on the user's screen.

[0841] Specific behavior:

[0842] The server sends the generated chat messages, such as "Interesting!" and "Amazing, I'd love to hear more!", to the device. The device (participant's PC or smartphone) displays the chat messages on the user's screen. All participants can view the messages in real time through the chat window.

[0843] (Application example 2)

[0844] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0845] In online study groups and lectures, there is a lack of interactive communication between speakers and participants, so more active and meaningful communication is required. In addition, there is a challenge in providing appropriate feedback on audience emotions and reactions to improve the quality of real-time interactions.

[0846] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting the speaker's voice, means for converting the collected voice into text data, means for analyzing the context of the text data and selecting appropriate backchannels, means for sending the selected backchannels as chat messages, means for displaying the chat messages on a screen, means for collecting viewer comments and analyzing the context and emotions of the comments, and means for generating and displaying appropriate backchannels and reactions to the viewer based on the analysis results. This allows for real-time exchange of emotion-based feedback between the speaker, participants, viewers, etc., enabling more fulfilling interactive communication.

[0847] A "speaker" is a person who speaks at an online study group or lecture.

[0848] "Means for collecting audio" refers to devices and methods for recording the speaker's speech and transmitting it as digital data to a server.

[0849] "Means for converting voice into text data" refers to the technology or method for analyzing collected voice data and converting it into text format.

[0850] "Means for analyzing the context of text data" refers to natural language processing techniques and methods for analyzing text data and understanding its content and meaning.

[0851] "Means for selecting backchannels" refers to techniques and methods for selecting appropriate backchannels and responses based on the results of context analysis and emotion analysis.

[0852] "Means for sending as a chat message" refers to a technique or method for sending the generated backchannels or responses as an electronic message.

[0853] The "means for displaying chat messages on a screen" refers to a technique or method for displaying the sent messages on a display device.

[0854] "Means for collecting viewer comments" refers to techniques and methods for collecting comments from viewers in real time.

[0855] "Means for analyzing the context and sentiment of comments" refers to natural language processing and sentiment analysis technologies that analyze comments from viewers and understand their content and emotional state.

[0856] "Means for generating appropriate responses and reactions to viewers based on the analysis results" refers to technology or methods that automatically generate appropriate responses and reactions to viewers' comments based on the analysis results.

[0857] This invention provides a system for promoting interactive communication between a lecturer and an audience in an online study group or lecture meeting. The operation of this system will be described in detail below.

[0858] 1. Audio collection and conversion

[0859] A device with a microphone is used to collect the speaker's voice. The user (speaker) starts their lecture or explanation, and the device collects the speaker's voice in real time and sends this audio data in digital format to a server. The server then converts the received audio data into text data using a speech recognition engine (e.g., Google Speech-to-Text API).

[0860] 2. Context Analysis and Backchannel Generation

[0861] The server passes the text data to a natural language processing (NLP) module (such as SpaCy or NLTK) to analyze the context. The NLP module analyzes the context of the text data and determines appropriate backchannel phrases (e.g., "Uh-huh," "I see," "Great!", "Amazing!"). Based on the analysis results, the server generates appropriate backchannels.

[0862] 3. Collection and analysis of viewer comments

[0863] The server uses real-time communication technology (e.g., WebSocket) to collect comments from viewers in real time. After collecting the comments, these comments are also passed to the NLP module to analyze the context and sentiment. For sentiment analysis, an emotion engine (e.g., IBM Watson Tone Analyzer) is used.

[0864] 4. Reaction generation and display

[0865] The server generates appropriate responses and reactions for the audience based on the analysis results. These responses and reactions are sent as chat messages, which are then displayed on the screens of the user (speaker) and audience members. This allows the speaker to see audience reactions in real time, and audience members can receive feedback on their own comments.

[0866] Specific examples

[0867] Below is a specific example scenario of a live stream where a speaker is talking about protecting the environment.

[0868] 1. Speaker: "We are being asked to reduce our use of plastic by 50% over the next 10 years."

[0869] 2. Real-time analysis and backchannel generation:

[0870] The speech recognition engine converts speech into text.

[0871] The NLP module analyzes the context and generates an appropriate response: "That's important!"

[0872] The sentiment analysis engine analyzes and emphasizes the speaker's passion, adding "What an amazing initiative!"

[0873] 3. Feedback display:

[0874] The speaker sees these responses in real time and continues speaking with even more enthusiasm.

[0875] Prompt Sentence Examples

[0876] Below is an example of a prompt sentence that is input to a generative AI model.

[0877] Speaker's audio data

[0878] There is a call to reduce plastic use by 50% over the next 10 years.

[0879] prompt

[0880] Generate appropriate responses to what this speaker has said based on audience sentiment analysis.

[0881] Generated responses (example)

[0882] That's important!

[0883] What an amazing initiative!

[0884] This allows for real-time emotional feedback between the speaker and the audience, enabling richer, more interactive communication.

[0885] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0886] Step 1:

[0887] The speaker (user) starts a lecture or explanation using a device with a microphone. The device collects the speaker's voice in real time, converts this voice data into a digital format, and sends it to the server.

[0888] Input: Speaker's audio data

[0889] Output: Digital audio data

[0890] How it works: The device collects audio through a microphone, converts it into digital form, and sends it over the network to a server.

[0891] Step 2:

[0892] The server passes the received voice data to a voice recognition engine (for example, Google Speech-to-Text API) and converts the voice data into text data.

[0893] Input: Digital audio data

[0894] Output: Text data

[0895] Specific operation: The server sends audio data to the Google Speech-to-Text API and receives the converted text data.

[0896] Step 3:

[0897] The server passes the text data to a natural language processing (NLP) module (e.g., SpaCy or NLTK) for context analysis, which then determines the appropriate backchannel phrases.

[0898] Input: Text data

[0899] Output: Context analysis results and backchannel phrases

[0900] What it does: The server passes the text data to the NLP module, which generates appropriate back-channel responses (e.g., "Uh-huh," "I see," "Great!").

[0901] Step 4:

[0902] The server formats the generated backchannel phrases as chat messages and sends them to the terminal.

[0903] Input: Backchannel phrase

[0904] Output: Chat message

[0905] Specific operation: The server converts the generated backchannel phrases into chat messages and sends them to each viewer's device.

[0906] Step 5:

[0907] The terminal displays the received chat messages on the screen so that the speaker and the audience can see the messages.

[0908] Input: Chat message

[0909] Output: Chat messages displayed

[0910] Specific operation: The device displays the received chat message on the user's screen.

[0911] Step 6:

[0912] The server collects comments from viewers in real time and passes these comments to a natural language processing (NLP) module for context and sentiment analysis.

[0913] Input: Viewer comments

[0914] Output: Contextual and sentiment analysis results

[0915] Specific operation: The server collects comments using real-time communication technology (such as WebSocket) and passes them to the NLP module and emotion engine for analysis.

[0916] Step 7:

[0917] The server generates appropriate responses and reactions based on the viewers' comments, which are then formatted as chat messages and sent to the device.

[0918] Input: Contextual and sentiment analysis results

[0919] Output: Chat message

[0920] Specific operation: The server generates responses and reactions to the comments based on the analysis results, formats them as chat messages, and sends them.

[0921] Step 8:

[0922] The device displays the generated responses and reactions on the viewer's screen, allowing them to see the feedback in real time.

[0923] Input: Chat message

[0924] Output: The displayed responses and reactions

[0925] What it does: The device displays the received responses and reactions on the viewer's screen, providing real-time feedback.

[0926] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0927] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0928] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0929] [Third embodiment]

[0930] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0931] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0932] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0933] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0934] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0935] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0936] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0937] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0938] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0939] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0940] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0941] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0942] This invention is a system for realizing interactive communication between speakers and participants in online study groups and lectures. Specifically, it collects the speaker's voice, converts that voice into text, analyzes the context of the text, and automatically generates and transmits appropriate responses.

[0943] This system mainly consists of the following devices and technologies:

[0944] 1. Collecting speaker audio

[0945] The user (speaker) uses a device with a microphone (computer, smartphone, tablet, etc.) to give a lecture or explanation.

[0946] The device collects the speaker's voice in real time and transmits it to the server as digital audio data.

[0947] 2. Converting voice data to text

[0948] The server converts the received voice data into text data using speech recognition technology. This process uses a speech recognition engine (such as the Google Speech-to-Text API).

[0949] The server receives the converted text data and prepares it for contextual analysis.

[0950] 3. Context Analysis and Backchannel Generation

[0951] The server uses natural language processing technology to analyze the context of the text data, identifying what the speaker said and when, and then selects appropriate back-channel responses (e.g., "Uh-huh," "I see," "Great!", "Amazing!", etc.).

[0952] The server automatically generates the selected responses in the form of chat messages.

[0953] 4. Sending and Viewing Chat Messages

[0954] The server transmits the generated backchannel message to the terminal.

[0955] The terminal displays the sent chat message on the user's screen so that all participants can see the message.

[0956] Specific examples

[0957] Example 1: A speaker is explaining a theory

[0958] 1. Audio collection

[0959] User: "Based on this theory, performance will improve significantly."

[0960] Device: Collects audio from the microphone and sends it to the server.

[0961] 2. Text Conversion

[0962] Server: Sends the speech data to the speech recognition engine and receives the text data, "Based on this theory, performance will be significantly improved."

[0963] 3. Context Analysis and Backchannel Generation

[0964] Server: Analyzes the text data using natural language processing technology and selects the appropriate "Like!" response phrase based on the context.

[0965] Server: Generate a "Like" chat message.

[0966] 4. Chat sending and display

[0967] Server: Sends the generated "like" chat message to the device.

[0968] Terminal: Chat messages are displayed on the user's screen.

[0969] In this way, appropriate responses are automatically inserted while the speaker is speaking, making it easier for participants to actively engage with the content of the lecture, and the speaker can also see participants' reactions in real time. This system improves the quality of online communication and provides a more effective learning environment.

[0970] The processing flow will be explained below.

[0971] Step 1:

[0972] The user accesses the online study group link and enters their login information.

[0973] The terminal sends the entered login information to the server.

[0974] The server checks the authentication information against a database to verify that the participant is legitimate.

[0975] Step 2:

[0976] The user (speaker) uses a device with a microphone to begin a lecture or explanation.

[0977] The device collects the speaker's audio in real time.

[0978] The device transmits the collected voice data to the server in real time.

[0979] Step 3:

[0980] The server sends the received voice data to the voice recognition engine.

[0981] A voice recognition engine converts the voice data into text data.

[0982] The server receives the converted text data.

[0983] Step 4:

[0984] The server passes the text data to a natural language processing (NLP) module.

[0985] The NLP module analyzes the context of the text data and determines appropriate responses.

[0986] The server selects appropriate response phrases based on the analysis results.

[0987] Step 5:

[0988] The server generates the selected backchannel as a chat message.

[0989] The server sends the generated chat message to the device.

[0990] The terminal displays the chat messages on the user's screen.

[0991] Step 6:

[0992] Users (participants) check the chat messages and enter their reactions or questions in the chat field as necessary.

[0993] The device sends the input responses and questions to the server.

[0994] The server collects participants' responses and questions and uses them to generate subsequent responses and analyze context.

[0995] Example 1

[0996] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0997] In online study groups and lectures, there is a tendency for there to be a lack of interaction between the speaker and the participants, which can lead to a decline in the quality of communication. This makes it difficult for the speaker to get participants' reactions in real time, and for participants to maintain interest in the content of the lecture.

[0998] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0999] In this invention, the server includes means for collecting the speaker's voice, means for converting the collected voice into text data, means for analyzing the context of the text data and selecting appropriate backchannels, means for transmitting the selected backchannels as chat messages, and means for displaying the chat messages on a screen. This allows the speaker to receive appropriate backchannels from the participants in real time, and allows the participants to maintain their interest in the content of the lecture.

[1000] "Means for collecting the speaker's voice" refers to a device or function that uses the microphone of the device (computer, smartphone, tablet, etc.) used by the speaker to collect the voice during the lecture as digital voice data in real time.

[1001] "Means for converting collected voice data into text data" refers to technology that uses a voice recognition engine to convert collected digital voice data into text data. For example, Google Speech-to-Text API is one such technology.

[1002] "Means for analyzing the context of text data and selecting appropriate backchannels" refers to technologies and algorithms for analyzing the content and context of converted text data using natural language processing technology and selecting appropriate backchannel phrases.

[1003] "Means for transmitting the selected backchannels as chat messages" refers to technologies and protocols for converting the selected backchannel phrases into chat message format and transmitting them to participants' devices. Examples include WebSocket and HTTP.

[1004] "Means for displaying chat messages on a screen" refers to software and technology for displaying chat messages sent to participants' devices on an interface in real time. For example, this includes online conference tools and dedicated applications.

[1005] "Speech recognition technology" refers to technology for converting voice data into text data, including voice models and machine learning algorithms. An example is the Google Speech-to-Text API.

[1006] "Natural language processing technology" is a technology for analyzing the content and context of text data and understanding its meaning, and includes technologies such as morphological analysis, semantic analysis, and sentiment analysis. Models such as BERT are also part of this technology.

[1007] This invention is a system for realizing interactive communication between speakers and participants in online study groups and lectures. This system collects the speaker's voice, converts the voice into text, analyzes the context of the text, and automatically generates and transmits appropriate responses. The specific configuration and implementation method of this system are described in detail below.

[1008] Hardware and Software Use

[1009] 1. Audio collection

[1010] The user (speaker) gives a lecture or explanation using a device with a microphone (computer, smartphone, tablet, etc.).

[1011] The device uses a microphone to collect the speaker's voice in real time and uses an internet connection to send it to a server as digital audio data.

[1012] 2. Text Conversion

[1013] The server converts the received voice data into text data using speech recognition technology, using a speech recognition engine such as the Google Speech-to-Text API.

[1014] The server receives the converted text data and prepares it for contextual analysis.

[1015] 3. Context Analysis and Backchannel Generation

[1016] The server analyzes the context of the text data using natural language processing (NLP) techniques, including morphological analysis, semantic analysis, and sentiment analysis, and also uses AI models such as BERT.

[1017] The server selects appropriate backchannel phrases based on the analysis results. The selected backchannels are either chosen from a predefined list of phrases or dynamically generated by AI.

[1018] 4. Sending and Viewing Chat Messages

[1019] The server sends the generated backchannel message to the device using a communication protocol such as WebSocket or HTTP.

[1020] The device displays the sent chat messages on the user's screen using an online conference tool or a dedicated application.

[1021] Specific examples

[1022] For example, here is the flow when a speaker explains, "Based on this theory, performance will improve significantly."

[1023] 1. Audio collection

[1024] User: "Based on this theory, performance will improve significantly."

[1025] Device: Collects audio from the microphone and sends it to the server.

[1026] 2. Text Conversion

[1027] Server: Sends audio data to the Google Speech-to-Text API and receives text data that says, "Based on this theory, performance will be significantly improved."

[1028] 3. Context Analysis and Backchannel Generation

[1029] Server: Analyzes the text data using natural language processing technology and selects the appropriate "Like!" response phrase based on the context.

[1030] Server: Generate a "Like" chat message.

[1031] 4. Sending and Viewing Chat Messages

[1032] Server: Sends the generated "like" chat message to the device.

[1033] Terminal: A web application displays chat messages on the user's screen.

[1034] This system allows users to receive appropriate responses from participants in real time during a lecture, and also helps participants to become more interested in the content of the lecture, thereby improving the quality of online communication and providing a more effective learning environment.

[1035] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1036] Step 1: Audio Collection

[1037] Input: A user speaks using the microphone on their device (computer, smartphone, tablet, etc.).

[1038] What happens: The speaker gives a presentation or talk, and the microphone collects the audio.

[1039] Device operation: The device's microphone collects the speaker's voice in real time.

[1040] Output: Collected digital audio data is generated and used in the next processing step.

[1041] Step 2: Sending audio data

[1042] Input: Digital audio data collected by the device.

[1043] What happens next: Your device sends audio data to a server over your internet connection.

[1044] Device operation: The device prepares the collected digital audio data and initiates communication to send it to the server.

[1045] Output: Audio data is sent to and received by the server.

[1046] Step 3: Convert audio data to text

[1047] Input: Digital audio data received by the server.

[1048] Specific actions taken: The server uses speech recognition technology (e.g., Google Speech-to-Text API) to convert the voice data into text data.

[1049] Server operation: The server sends the voice data to the speech recognition engine, which analyzes the voice and generates text data.

[1050] Output: The audio data is converted into "text data" and used in the next context analysis step.

[1051] Step 4: Contextual analysis of text data

[1052] Input: Text data returned from the speech recognition engine.

[1053] Specific actions taken: The server uses natural language processing techniques to analyze the context of the text data, including morphological analysis, semantic analysis, and sentiment analysis.

[1054] Server operation: The server analyzes the text data, performs calculations to identify the content and timing of the speaker's remarks, and extracts key keywords and sentiment.

[1055] Output: Information based on the contextual analysis is generated and used in the next step of backchannel generation.

[1056] Step 5: Selecting and generating backchannel phrases

[1057] Input: Information based on contextual analysis.

[1058] What happens: The server selects appropriate backchannels based on the contextual analysis results. The selected backchannels are either chosen from a pre-defined list or dynamically generated by a generative AI model.

[1059] Server operation: The server selects appropriate backchannel phrases based on the analysis results and automatically generates them in the form of chat messages.

[1060] Output: The selected backchannel phrase is generated as a chat message and used in the next sending step.

[1061] Step 6: Send a chat message

[1062] Input: An automatically generated chat backchannel message.

[1063] Specific actions taken: The server sends chat messages over the Internet to the user's device.

[1064] Server operation: The server initiates communication to send the generated chat message to the specified device. WebSocket or HTTP is used as the communication protocol.

[1065] Output: The chat message is sent to the device.

[1066] Step 7: View chat messages

[1067] Input: The chat message sent by the server.

[1068] Specific actions taken: Display chat messages received by the device on the screen.

[1069] Terminal Action: Displays a new message in a chat message window or similar, allowing all participants to see the message in real time.

[1070] Output: A backchannel message will be displayed in the chat window.

[1071] This series of processes allows users to receive appropriate responses from participants in real time during a lecture, making it easier for participants to become interested in the content of the lecture. This improves the quality of online communication and provides a more effective learning environment.

[1072] (Application example 1)

[1073] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1074] In online shopping virtual stores, customers often have difficulty getting a proper response in real time when they have questions or comments about products, which can lead to a poor customer experience and reduced purchasing intent.

[1075] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1076] In this invention, the server includes means for collecting the speaker's voice, means for converting the collected voice into text data, means for analyzing the context of the text data and selecting appropriate backchannels, means for sending the selected backchannels as chat messages, means for displaying the chat messages on a screen, means for collecting customer voice, converting it into text, analyzing the context and generating a response, and means for displaying the generated response message on a display device, thereby improving interaction with customers in real time and enabling a better shopping experience.

[1077] The "means for collecting the speaker's voice" refers to a device or method for collecting the speaker's voice in real time using a microphone device or the like and converting it into digital voice data.

[1078] The "means for converting collected voice data into text data" refers to a device or method for converting collected voice data into text information using voice recognition technology.

[1079] "Means for analyzing the context of text data and selecting appropriate responses" refers to a device or method that uses natural language processing technology to analyze the content and timing of statements contained in text data, and generates or selects appropriate responses or comments based on that context.

[1080] The "means for transmitting the selected backchannel as a chat message" refers to a device or method for transmitting the selected backchannel or response message in a chat format via a network.

[1081] The "means for displaying chat messages on a screen" refers to a device or method for displaying received chat messages on a user interface so that the user can check their contents.

[1082] "Means for collecting customer speech, converting it into text, analyzing the context, and generating a response" refers to a device or method that collects speech uttered by a customer, converts it into text information, analyzes the context, and automatically generates an appropriate response.

[1083] "Means for displaying the generated response message on a display device" refers to a device or method that outputs the generated response message to a display or other display device so that the user can visually confirm the message.

[1084] This invention is a system that automatically supports interactive communication in online lectures and virtual stores using speech recognition and context analysis. This system collects user speech, converts it into text in real time, analyzes the context, generates appropriate responses, and displays them on a display device. To function effectively, this system requires the following hardware and software:

[1085] Hardware and Software

[1086] Microphone devices: Used to collect audio data. This includes computers, smartphones, tablets, etc. with microphones.

[1087] Computer server: Receives the voice data and analyzes it using speech recognition and natural language processing technologies. It converts the voice data into text using Google Cloud's Speech-to-Text API and applies the GPT-3.5-turbo model using the transformers library.

[1088] Display device: A display device that shows chat messages and reply messages to users.

[1089] Data processing flow

[1090] 1. Audio collection: The device (user's smartphone or computer) collects the user's audio through a microphone device and transmits it to the server as digital audio data.

[1091] 2. Converting speech data to text: The server converts the speech data received into text using Google Cloud's Speech-to-Text API and prepares it for contextual analysis.

[1092] 3. Contextual analysis and response generation: The server uses natural language processing techniques to analyze the context of the text data, and uses the GPT-3.5-turbo model to generate appropriate responses based on that analysis.

[1093] 4. Creating and displaying a chat message or response message: The server sends the generated response to the terminal, which displays the message on its display device.

[1094] Specific examples

[1095] Example 1: Virtual store example

[1096] 1. Audio collection:

[1097] Customer: "Do you have this jacket in any other colors?"

[1098] Device: Collects audio from the microphone and sends it to the server.

[1099] 2. Text conversion:

[1100] Server: Sends the voice data to the voice recognition engine and receives the text data "Do you have this jacket in other colors?"

[1101] 3. Context analysis and response generation:

[1102] Server: Analyzes the text data using natural language processing technology and generates a response based on the context: "Of course we have it. Currently, it comes in three colors: black, blue, and red."

[1103] 4. Chat sending and display:

[1104] Server: Sends the generated response message to the terminal.

[1105] Terminal: Displays the response message on the user's screen.

[1106] Example prompt sentence:

[1107] Customer: Do you have this jacket in other colors?

[1108] context:

[1109] Generate a response.

[1110] The present invention improves customer experience in virtual stores and enables real-time interactive communication.

[1111] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1112] Step 1:

[1113] Audio collection: The device (user's smartphone or computer) collects the user's voice using a microphone device. When the user speaks, the microphone captures it and converts it into digital audio data in real time. This digital audio data is sent to the next processing step.

[1114] Input: User's voice

[1115] Output: Digital audio data

[1116] What it does: The microphone captures audio and the application converts it into a digital format.

[1117] Step 2:

[1118] Speech-to-text conversion: The device sends the collected digital speech data to a server, which converts the speech to text using Google Cloud's Speech-to-Text API and prepares the converted text for analysis.

[1119] Input: Digital audio data

[1120] Output: Text data

[1121] What happens: The server sends the audio data to the Google Cloud Speech-to-Text API, which converts the audio into text.

[1122] Step 3:

[1123] Contextual analysis and response generation: The server receives the text data and analyzes its context using natural language processing techniques. This analysis uses the transformers library and the GPT-3.5-turbo model. The generative AI model generates an appropriate response prompt from the analysis results.

[1124] Input: Text data

[1125] Output: Response message

[1126] Specific operation: The server sends a prompt to the GPT-3.5-turbo model using text data as input, and the model generates a response message.

[1127] Step 4:

[1128] Sending a response message: The server sends the generated response message to the terminal, which receives it and prepares it for display on the display device.

[1129] Input: Response message

[1130] Output: Response message sent to the terminal

[1131] Specific operation: The server sends a response message to the terminal over the network.

[1132] Step 5:

[1133] Display: The terminal displays the received response message on a display device, allowing the user to see the generated response in real time.

[1134] Input: Response message sent to the terminal

[1135] Output: Response message displayed on display device

[1136] Specific actions: The terminal renders the response message on the screen and displays it on the display device.

[1137] This enables real-time two-way communication between users and the system, improving the customer experience in the virtual store.

[1138] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1139] This invention is a system for promoting interactive communication between speakers and participants in online study groups and lectures. In particular, it combines an emotion engine to analyze the user's emotions and generate and display appropriate responses, enabling more lively communication.

[1140] This system is composed of the following major devices and technologies:

[1141] 1. Collecting speaker audio

[1142] The user (speaker) begins a lecture or explanation using a device with a microphone.

[1143] The terminal collects the speaker's voice in real time and transmits it to the server as digital voice data.

[1144] 2. Converting voice data to text

[1145] The server transmits the received voice data to a voice recognition engine, which converts the voice data into text data.

[1146] The speech recognition engine returns the converted text data to the server, which receives it.

[1147] 3. Context Analysis and Backchannel Generation

[1148] The server passes the text data to a natural language processing (NLP) module for contextual analysis.

[1149] The NLP module analyzes the context of the text data and determines appropriate interjections (e.g., "Uh-huh," "I see," "Great!", "Amazing!", etc.).

[1150] The server selects and generates appropriate responses based on the analysis results.

[1151] 4. Sentiment analysis and backchannel optimization

[1152] The server passes the collected voice data and text data to the emotion engine.

[1153] The emotion engine analyzes emotions from the user's voice and text, and selects appropriate responses based on the results.

[1154] For example, through emotion analysis, if a speaker is speaking enthusiastically, the system will generate many positive responses such as "Amazing!", but if the listeners are feeling stressed, it will generate neutral or empathetic responses such as "I see" or "Did you understand?"

[1155] 5. Sending and Viewing Chat Messages

[1156] The server sends the generated chat message to the device.

[1157] The terminal displays the sent chat message on the user's screen so that all participants can see the message.

[1158] Specific examples

[1159] Example 1: A speaker is explaining a new theory

[1160] 1. Audio collection

[1161] User (speaker): "Based on this theory, performance will improve significantly."

[1162] Device: Collects audio from the microphone and sends it to the server.

[1163] 2. Text Conversion

[1164] Server: Sends the speech data to the speech recognition engine and receives the text data, "Based on this theory, performance will be significantly improved."

[1165] 3. Context Analysis and Backchannel Generation

[1166] Server: Analyzes the text data using natural language processing technology and selects a "Like!" response based on the context.

[1167] Server: Generate a "Like" chat message.

[1168] 4. Sentiment analysis and backchannel optimization

[1169] Server: Passes voice and text data to the emotion engine.

[1170] Emotion Engine: If a speaker is speaking confidently, it analyzes positive emotions and selects additional back-channel responses like "Awesome!" to emphasize them.

[1171] Server: Also generates a "Awesome!" chat message.

[1172] 5. Chat sending and display

[1173] Server: Sends the generated "Like" and "Awesome" chat messages to the device.

[1174] Terminal: Chat messages are displayed on the user's screen.

[1175] This system will increase interactivity between speakers and participants in online study groups and lectures, and provide emotion-based feedback, resulting in more meaningful and effective communication.

[1176] The processing flow will be explained below.

[1177] Program processing steps

[1178] Step 1:

[1179] The user accesses the online study group link and enters their login information.

[1180] The terminal sends the entered login information to the server.

[1181] The server checks the authentication information against a database to verify that the participant is legitimate.

[1182] Step 2:

[1183] The user (speaker) begins speaking using a device with a microphone.

[1184] The device collects the speaker's voice in real time and transmits it to the server as digital audio data.

[1185] Step 3:

[1186] The server sends the received voice data to the voice recognition engine.

[1187] A voice recognition engine converts the voice data into text data.

[1188] The server receives the converted text data.

[1189] Step 4:

[1190] The server passes the text data to a natural language processing (NLP) module.

[1191] The NLP module analyzes the context of the text data and determines appropriate backchannel phrases.

[1192] The server selects the appropriate response based on the analysis results.

[1193] Step 5:

[1194] The server sends the collected voice data and converted text data to the emotion engine.

[1195] The emotion engine analyzes the user's emotions and selects the most appropriate response phrase based on the results.

[1196] The server generates the selected backchannel phrases and constructs them as chat messages.

[1197] Step 6:

[1198] The server sends the generated chat message to the device.

[1199] The terminal displays the sent chat message on the user's screen.

[1200] Specific examples

[1201] Example 1: A speaker is explaining a new theory

[1202] Step 1:

[1203] The user (speaker) says, "Based on this theory, performance will improve significantly."

[1204] The terminal collects the speaker's voice and sends it to the server.

[1205] Step 2:

[1206] The server receives the voice data and sends it to the voice recognition engine.

[1207] The speech recognition engine converts the speech into text and generates text data that says, "Based on this theory, performance will be significantly improved."

[1208] The server receives the conversion result.

[1209] Step 3:

[1210] The server passes the text data to the NLP module.

[1211] The NLP module analyzes the context and determines the "like" response.

[1212] The server selects "Likes" based on the analysis results.

[1213] Step 4:

[1214] The server sends the text data and voice data to the emotion engine.

[1215] The emotion engine analyzes the user's emotions and detects positive emotions.

[1216] The emotion engine also determines that the response "Amazing!" is appropriate and selects it.

[1217] The server generates "Like" and "Awesome" chat messages.

[1218] Step 5:

[1219] The server sends the generated chat message to the device.

[1220] The terminal displays the chat messages on the user's screen.

[1221] As shown in this example, the system converts the speaker's speech into text and generates and displays appropriate responses using natural language processing and sentiment analysis. This process improves interaction in online study groups and lectures, promoting two-way communication between speakers and participants.

[1222] Example 2

[1223] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1224] Online study groups and lectures face a lack of interactive communication between speakers and participants. In particular, there is a lack of a way to automatically generate appropriate responses and provide emotion-based feedback to the speaker. This makes it difficult for participants to respond appropriately to the content of the lecture, and the lecture tends to become a one-way communication of information.

[1225] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1226] In this invention, the server includes means for collecting the speaker's voice, means for converting the collected voice into text data, means for analyzing the context of the text data and selecting appropriate backchannels, means for optimizing the selected backchannels based on emotion analysis, means for sending the optimized backchannels as chat messages, and means for displaying the chat messages on a screen. This enables interactive communication with participants in online study groups and lectures by automatically generating appropriate backchannels based on emotion.

[1227] "Audio collection means" refers to a device or technology that captures the speaker's voice as digital data and transmits that data to a server.

[1228] The "means for converting voice into text data" refers to a voice recognition technology or system that automatically converts collected voice data into text data.

[1229] "Means for analyzing context and selecting appropriate backchannels" refers to natural language processing (NLP) technology or systems that analyze the content of text data and determine appropriate backchannel phrases.

[1230] The "means for optimizing based on emotion analysis" is an emotion engine or algorithm that analyzes the user's emotions and selects and optimizes appropriate responses based on the results.

[1231] The "means for transmitting backchannels as chat messages" refers to a communication technology or system that transmits the generated backchannels to a terminal in the form of a chat message.

[1232] The "means for displaying chat messages on a screen" refers to a technology or system for displaying sent chat messages on the terminals of participants.

[1233] This invention is a system for promoting interactive communication between speakers and participants in online study groups and lectures. In particular, by combining an emotion engine to analyze the user's emotions and generate and display appropriate responses, more lively communication is possible.

[1234] Hardware and software used

[1235] 1. A device with a microphone (e.g., a headset or smartphone)

[1236] 2. Server (including speech recognition engine, natural language processing engine, emotion engine)

[1237] 3. Devices (PC, smartphone, tablet)

[1238] Detailed explanation of the process

[1239] First, the user (speaker) begins their lecture using a device with a microphone. The device collects the speaker's voice in real time and transmits it to the server as digital audio data. For example, the speaker's utterance, "Good morning, today I will talk about the future of AI," is collected by the microphone and transmitted as digital data to the server.

[1240] Next, the server sends the received voice data to a speech recognition engine and converts the voice data into text data. Specifically, it uses the Google Cloud Speech-to-Text API to convert the voice data into text data such as "Good morning, today I'll talk about the future of AI." The server receives this converted text data.

[1241] The server then passes this text data to a natural language processing (NLP) module for context analysis. Using Hugging Face's Transformers library, the server analyzes the context of the text data and determines appropriate backchannel phrases. For example, it selects "Interesting!" from the context "Today we'll talk about the future of AI."

[1242] The server then passes the collected voice and text data to the emotion engine. Using IBM Watson's Tone Analyzer, the engine analyzes the user's emotions and selects the most appropriate backchannel based on the results. For example, if the speaker is analyzed as speaking with confidence, the engine selects an additional positive backchannel such as "That's great! I'd like to hear more!" By sending this backchannel, it becomes possible to provide feedback based on the speaker's emotions.

[1243] In the final stage, the server sends the generated chat message to the device, which displays it on the user's screen so that all participants can see it. For example, chat messages such as "Interesting!" and "Awesome, I'd love to hear more!" are displayed and shared with participants in real time.

[1244] Specific examples

[1245] Example: A speaker is explaining a new theory.

[1246] 1. User (speaker): "Based on this theory, performance will improve significantly."

[1247] 2. Device: Collects audio from the microphone and sends it to the server.

[1248] 3. Server: Sends the audio data to the Google Cloud Speech-to-Text API and receives it as text data.

[1249] 4. Server: The text data is analyzed using Hugging Face's Transformers library to determine whether it is a "Like!"

[1250] 5. Server: Use IBM Watson's Tone Analyzer to analyze emotions and select additional "Amazing!" responses.

[1251] 6. Server: Sends "Like" and "Awesome" chat messages to the device.

[1252] 7. Terminal: Chat messages are displayed on the user's screen so that participants can see them.

[1253] Prompt Sentence Examples

[1254] The speaker will talk about "The Future of AI Technology." Part of the lecture will include the following:

[1255] Speaker remarks: "These technologies have the potential to profoundly change the way we live our lives."

[1256] System-based text conversion: "These technologies have the potential to significantly change our lives."

[1257] System analysis: "Great!"

[1258] Sentiment analysis by the system: "Amazing!"

[1259] Expected output:

[1260] When the speaker says, "These technologies have the potential to dramatically change our lives," the responses "Amazing!" and "Amazing!" are displayed.

[1261] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1262] Step 1:

[1263] Collecting speaker audio

[1264] User (speaker): Starts speaking using a device with a microphone.

[1265] Input: Speaker audio from a device with a microphone.

[1266] Output: Audio data in digital format.

[1267] Specific behavior:

[1268] The speaker says, "Good morning, today I'll be talking about the future of AI." The device (computer or smartphone app) captures the voice with its built-in microphone, converts the voice data into digital format, and transmits it to a server in real time.

[1269] Step 2:

[1270] Converting audio data to text

[1271] Server: Sends the received voice data to the speech recognition engine.

[1272] Input: Audio data in digital format.

[1273] Output: Text data.

[1274] Specific behavior:

[1275] The server sends the received voice data to the Google Cloud Speech-to-Text API and converts it into text data such as "Good morning, today I'll talk about the future of AI." The converted text data is then received by the server.

[1276] Step 3:

[1277] Context analysis and backchannel generation

[1278] Server: Passes the text data to a natural language processing (NLP) module for contextual analysis.

[1279] Input: The converted text data.

[1280] Output: An appropriate backchannel phrase.

[1281] Specific behavior:

[1282] The server passes the text data to an NLP module using Hugging Face's Transformers library. The NLP module analyzes the context, "Today we'll talk about the future of AI," and selects an appropriate back-channel phrase, such as "Interesting!" The server then generates a chat message saying, "Interesting!"

[1283] Step 4:

[1284] Sentiment analysis and backchannel optimization

[1285] Server: Passes the collected voice and text data to the emotion engine.

[1286] Input: Audio and text data.

[1287] Output: Optimized backchannel phrases.

[1288] Specific behavior:

[1289] The server passes the audio and text data to IBM Watson's Tone Analyzer. If the emotion engine determines that the speaker is speaking confidently, it selects a positive response such as, "Great, I'd love to hear more!" The server then generates this new response as a chat message.

[1290] Step 5:

[1291] Sending and viewing chat messages

[1292] Server: Sends generated chat messages to devices.

[1293] Input: Optimized backchannel phrase.

[1294] Output: The chat messages that are displayed on the user's screen.

[1295] Specific behavior:

[1296] The server sends the generated chat messages, such as "Interesting!" and "Amazing, I'd love to hear more!", to the device. The device (participant's PC or smartphone) displays the chat messages on the user's screen. All participants can view the messages in real time through the chat window.

[1297] (Application example 2)

[1298] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1299] In online study groups and lectures, there is a lack of interactive communication between speakers and participants, so more active and meaningful communication is required. In addition, there is a challenge in providing appropriate feedback on audience emotions and reactions to improve the quality of real-time interactions.

[1300] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting the speaker's voice, means for converting the collected voice into text data, means for analyzing the context of the text data and selecting appropriate backchannels, means for sending the selected backchannels as chat messages, means for displaying the chat messages on a screen, means for collecting viewer comments and analyzing the context and emotions of the comments, and means for generating and displaying appropriate backchannels and reactions to the viewer based on the analysis results. This allows for real-time exchange of emotion-based feedback between the speaker, participants, viewers, etc., enabling more fulfilling interactive communication.

[1301] A "speaker" is a person who speaks at an online study group or lecture.

[1302] "Means for collecting audio" refers to devices and methods for recording the speaker's speech and transmitting it as digital data to a server.

[1303] "Means for converting voice into text data" refers to the technology or method for analyzing collected voice data and converting it into text format.

[1304] "Means for analyzing the context of text data" refers to natural language processing techniques and methods for analyzing text data and understanding its content and meaning.

[1305] "Means for selecting backchannels" refers to techniques and methods for selecting appropriate backchannels and responses based on the results of context analysis and emotion analysis.

[1306] "Means for sending as a chat message" refers to a technique or method for sending the generated backchannels or responses as an electronic message.

[1307] The "means for displaying chat messages on a screen" refers to a technique or method for displaying the sent messages on a display device.

[1308] "Means for collecting viewer comments" refers to techniques and methods for collecting comments from viewers in real time.

[1309] "Means for analyzing the context and sentiment of comments" refers to natural language processing and sentiment analysis technologies that analyze comments from viewers and understand their content and emotional state.

[1310] "Means for generating appropriate responses and reactions to viewers based on the analysis results" refers to technology or methods that automatically generate appropriate responses and reactions to viewers' comments based on the analysis results.

[1311] This invention provides a system for promoting interactive communication between a lecturer and an audience in an online study group or lecture meeting. The operation of this system will be described in detail below.

[1312] 1. Audio collection and conversion

[1313] A device with a microphone is used to collect the speaker's voice. The user (speaker) starts their lecture or explanation, and the device collects the speaker's voice in real time and sends this audio data in digital format to a server. The server then converts the received audio data into text data using a speech recognition engine (e.g., Google Speech-to-Text API).

[1314] 2. Context Analysis and Backchannel Generation

[1315] The server passes the text data to a natural language processing (NLP) module (such as SpaCy or NLTK) to analyze the context. The NLP module analyzes the context of the text data and determines appropriate backchannel phrases (e.g., "Uh-huh," "I see," "Great!", "Amazing!"). Based on the analysis results, the server generates appropriate backchannels.

[1316] 3. Collection and analysis of viewer comments

[1317] The server uses real-time communication technology (e.g., WebSocket) to collect comments from viewers in real time. After collecting the comments, these comments are also passed to the NLP module to analyze the context and sentiment. For sentiment analysis, an emotion engine (e.g., IBM Watson Tone Analyzer) is used.

[1318] 4. Reaction generation and display

[1319] The server generates appropriate responses and reactions for the audience based on the analysis results. These responses and reactions are sent as chat messages, which are then displayed on the screens of the user (speaker) and audience members. This allows the speaker to see audience reactions in real time, and audience members can receive feedback on their own comments.

[1320] Specific examples

[1321] Below is a specific example scenario of a live stream where a speaker is talking about protecting the environment.

[1322] 1. Speaker: "We are being asked to reduce our use of plastic by 50% over the next 10 years."

[1323] 2. Real-time analysis and backchannel generation:

[1324] The speech recognition engine converts speech into text.

[1325] The NLP module analyzes the context and generates an appropriate response: "That's important!"

[1326] The sentiment analysis engine analyzes and emphasizes the speaker's passion, adding "What an amazing initiative!"

[1327] 3. Feedback display:

[1328] The speaker sees these responses in real time and continues speaking with even more enthusiasm.

[1329] Prompt Sentence Examples

[1330] Below is an example of a prompt sentence that is input to a generative AI model.

[1331] Speaker's audio data

[1332] There is a call to reduce plastic use by 50% over the next 10 years.

[1333] prompt

[1334] Generate appropriate responses to what this speaker has said based on audience sentiment analysis.

[1335] Generated responses (example)

[1336] That's important!

[1337] What an amazing initiative!

[1338] This allows for real-time emotional feedback between the speaker and the audience, enabling richer, more interactive communication.

[1339] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1340] Step 1:

[1341] The speaker (user) starts a lecture or explanation using a device with a microphone. The device collects the speaker's voice in real time, converts this voice data into a digital format, and sends it to the server.

[1342] Input: Speaker's audio data

[1343] Output: Digital audio data

[1344] How it works: The device collects audio through a microphone, converts it into digital form, and sends it over the network to a server.

[1345] Step 2:

[1346] The server passes the received voice data to a voice recognition engine (for example, Google Speech-to-Text API) and converts the voice data into text data.

[1347] Input: Digital audio data

[1348] Output: Text data

[1349] Specific operation: The server sends audio data to the Google Speech-to-Text API and receives the converted text data.

[1350] Step 3:

[1351] The server passes the text data to a natural language processing (NLP) module (e.g., SpaCy or NLTK) for context analysis, which then determines the appropriate backchannel phrases.

[1352] Input: Text data

[1353] Output: Context analysis results and backchannel phrases

[1354] What it does: The server passes the text data to the NLP module, which generates appropriate back-channel responses (e.g., "Uh-huh," "I see," "Great!").

[1355] Step 4:

[1356] The server formats the generated backchannel phrases as chat messages and sends them to the terminal.

[1357] Input: Backchannel phrase

[1358] Output: Chat message

[1359] Specific operation: The server converts the generated backchannel phrases into chat messages and sends them to each viewer's device.

[1360] Step 5:

[1361] The terminal displays the received chat messages on the screen so that the speaker and the audience can see the messages.

[1362] Input: Chat message

[1363] Output: Chat messages displayed

[1364] Specific operation: The device displays the received chat message on the user's screen.

[1365] Step 6:

[1366] The server collects comments from viewers in real time and passes these comments to a natural language processing (NLP) module for context and sentiment analysis.

[1367] Input: Viewer comments

[1368] Output: Contextual and sentiment analysis results

[1369] Specific operation: The server collects comments using real-time communication technology (such as WebSocket) and passes them to the NLP module and emotion engine for analysis.

[1370] Step 7:

[1371] The server generates appropriate responses and reactions based on the viewers' comments, which are then formatted as chat messages and sent to the device.

[1372] Input: Contextual and sentiment analysis results

[1373] Output: Chat message

[1374] Specific operation: The server generates responses and reactions to the comments based on the analysis results, formats them as chat messages, and sends them.

[1375] Step 8:

[1376] The device displays the generated responses and reactions on the viewer's screen, allowing them to see the feedback in real time.

[1377] Input: Chat message

[1378] Output: The displayed responses and reactions

[1379] What it does: The device displays the received responses and reactions on the viewer's screen, providing real-time feedback.

[1380] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1381] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1382] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1383] [Fourth embodiment]

[1384] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1385] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1386] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1387] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1388] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1389] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1390] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1391] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1392] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1393] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1394] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1395] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1396] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1397] This invention is a system for realizing interactive communication between speakers and participants in online study groups and lectures. Specifically, it collects the speaker's voice, converts that voice into text, analyzes the context of the text, and automatically generates and transmits appropriate responses.

[1398] This system mainly consists of the following devices and technologies:

[1399] 1. Collecting speaker audio

[1400] The user (speaker) uses a device with a microphone (computer, smartphone, tablet, etc.) to give a lecture or explanation.

[1401] The device collects the speaker's voice in real time and transmits it to the server as digital audio data.

[1402] 2. Converting voice data to text

[1403] The server converts the received voice data into text data using speech recognition technology. This process uses a speech recognition engine (such as the Google Speech-to-Text API).

[1404] The server receives the converted text data and prepares it for contextual analysis.

[1405] 3. Context Analysis and Backchannel Generation

[1406] The server uses natural language processing technology to analyze the context of the text data, identifying what the speaker said and when, and then selects appropriate back-channel responses (e.g., "Uh-huh," "I see," "Great!", "Amazing!", etc.).

[1407] The server automatically generates the selected responses in the form of chat messages.

[1408] 4. Sending and Viewing Chat Messages

[1409] The server transmits the generated backchannel message to the terminal.

[1410] The terminal displays the sent chat message on the user's screen so that all participants can see the message.

[1411] Specific examples

[1412] Example 1: A speaker is explaining a theory

[1413] 1. Audio collection

[1414] User: "Based on this theory, performance will improve significantly."

[1415] Device: Collects audio from the microphone and sends it to the server.

[1416] 2. Text Conversion

[1417] Server: Sends the speech data to the speech recognition engine and receives the text data, "Based on this theory, performance will be significantly improved."

[1418] 3. Context Analysis and Backchannel Generation

[1419] Server: Analyzes the text data using natural language processing technology and selects the appropriate "Like!" response phrase based on the context.

[1420] Server: Generate a "Like" chat message.

[1421] 4. Chat sending and display

[1422] Server: Sends the generated "like" chat message to the device.

[1423] Terminal: Chat messages are displayed on the user's screen.

[1424] In this way, appropriate responses are automatically inserted while the speaker is speaking, making it easier for participants to actively engage with the content of the lecture, and the speaker can also see participants' reactions in real time. This system improves the quality of online communication and provides a more effective learning environment.

[1425] The processing flow will be explained below.

[1426] Step 1:

[1427] The user accesses the online study group link and enters their login information.

[1428] The terminal sends the entered login information to the server.

[1429] The server checks the authentication information against a database to verify that the participant is legitimate.

[1430] Step 2:

[1431] The user (speaker) uses a device with a microphone to begin a lecture or explanation.

[1432] The device collects the speaker's audio in real time.

[1433] The device transmits the collected voice data to the server in real time.

[1434] Step 3:

[1435] The server sends the received voice data to the voice recognition engine.

[1436] A voice recognition engine converts the voice data into text data.

[1437] The server receives the converted text data.

[1438] Step 4:

[1439] The server passes the text data to a natural language processing (NLP) module.

[1440] The NLP module analyzes the context of the text data and determines appropriate responses.

[1441] The server selects appropriate response phrases based on the analysis results.

[1442] Step 5:

[1443] The server generates the selected backchannel as a chat message.

[1444] The server sends the generated chat message to the device.

[1445] The terminal displays the chat messages on the user's screen.

[1446] Step 6:

[1447] Users (participants) check the chat messages and enter their reactions or questions in the chat field as necessary.

[1448] The device sends the input responses and questions to the server.

[1449] The server collects participants' responses and questions and uses them to generate subsequent responses and analyze context.

[1450] Example 1

[1451] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1452] In online study groups and lectures, there is a tendency for there to be a lack of interaction between the speaker and the participants, which can lead to a decline in the quality of communication. This makes it difficult for the speaker to get participants' reactions in real time, and for participants to maintain interest in the content of the lecture.

[1453] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1454] In this invention, the server includes means for collecting the speaker's voice, means for converting the collected voice into text data, means for analyzing the context of the text data and selecting appropriate backchannels, means for transmitting the selected backchannels as chat messages, and means for displaying the chat messages on a screen. This allows the speaker to receive appropriate backchannels from the participants in real time, and allows the participants to maintain their interest in the content of the lecture.

[1455] "Means for collecting the speaker's voice" refers to a device or function that uses the microphone of the device (computer, smartphone, tablet, etc.) used by the speaker to collect the voice during the lecture as digital voice data in real time.

[1456] "Means for converting collected voice data into text data" refers to technology that uses a voice recognition engine to convert collected digital voice data into text data. For example, Google Speech-to-Text API is one such technology.

[1457] "Means for analyzing the context of text data and selecting appropriate backchannels" refers to technologies and algorithms for analyzing the content and context of converted text data using natural language processing technology and selecting appropriate backchannel phrases.

[1458] "Means for transmitting the selected backchannels as chat messages" refers to technologies and protocols for converting the selected backchannel phrases into chat message format and transmitting them to participants' devices. Examples include WebSocket and HTTP.

[1459] "Means for displaying chat messages on a screen" refers to software and technology for displaying chat messages sent to participants' devices on an interface in real time. For example, this includes online conference tools and dedicated applications.

[1460] "Speech recognition technology" refers to technology for converting voice data into text data, including voice models and machine learning algorithms. An example is the Google Speech-to-Text API.

[1461] "Natural language processing technology" is a technology for analyzing the content and context of text data and understanding its meaning, and includes technologies such as morphological analysis, semantic analysis, and sentiment analysis. Models such as BERT are also part of this technology.

[1462] This invention is a system for realizing interactive communication between speakers and participants in online study groups and lectures. This system collects the speaker's voice, converts the voice into text, analyzes the context of the text, and automatically generates and transmits appropriate responses. The specific configuration and implementation method of this system are described in detail below.

[1463] Hardware and Software Use

[1464] 1. Audio collection

[1465] The user (speaker) gives a lecture or explanation using a device with a microphone (computer, smartphone, tablet, etc.).

[1466] The device uses a microphone to collect the speaker's voice in real time and uses an internet connection to send it to a server as digital audio data.

[1467] 2. Text Conversion

[1468] The server converts the received voice data into text data using speech recognition technology, using a speech recognition engine such as the Google Speech-to-Text API.

[1469] The server receives the converted text data and prepares it for contextual analysis.

[1470] 3. Context Analysis and Backchannel Generation

[1471] The server analyzes the context of the text data using natural language processing (NLP) techniques, including morphological analysis, semantic analysis, and sentiment analysis, and also uses AI models such as BERT.

[1472] The server selects appropriate backchannel phrases based on the analysis results. The selected backchannels are either chosen from a predefined list of phrases or dynamically generated by AI.

[1473] 4. Sending and Viewing Chat Messages

[1474] The server sends the generated backchannel message to the device using a communication protocol such as WebSocket or HTTP.

[1475] The device displays the sent chat messages on the user's screen using an online conference tool or a dedicated application.

[1476] Specific examples

[1477] For example, here is the flow when a speaker explains, "Based on this theory, performance will improve significantly."

[1478] 1. Audio collection

[1479] User: "Based on this theory, performance will improve significantly."

[1480] Device: Collects audio from the microphone and sends it to the server.

[1481] 2. Text Conversion

[1482] Server: Sends audio data to the Google Speech-to-Text API and receives text data that says, "Based on this theory, performance will be significantly improved."

[1483] 3. Context Analysis and Backchannel Generation

[1484] Server: Analyzes the text data using natural language processing technology and selects the appropriate "Like!" response phrase based on the context.

[1485] Server: Generate a "Like" chat message.

[1486] 4. Sending and Viewing Chat Messages

[1487] Server: Sends the generated "like" chat message to the device.

[1488] Terminal: A web application displays chat messages on the user's screen.

[1489] This system allows users to receive appropriate responses from participants in real time during a lecture, and also helps participants to become more interested in the content of the lecture, thereby improving the quality of online communication and providing a more effective learning environment.

[1490] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1491] Step 1: Audio Collection

[1492] Input: A user speaks using the microphone on their device (computer, smartphone, tablet, etc.).

[1493] What happens: The speaker gives a presentation or talk, and the microphone collects the audio.

[1494] Device operation: The device's microphone collects the speaker's voice in real time.

[1495] Output: Collected digital audio data is generated and used in the next processing step.

[1496] Step 2: Sending audio data

[1497] Input: Digital audio data collected by the device.

[1498] What happens next: Your device sends audio data to a server over your internet connection.

[1499] Device operation: The device prepares the collected digital audio data and initiates communication to send it to the server.

[1500] Output: Audio data is sent to and received by the server.

[1501] Step 3: Convert audio data to text

[1502] Input: Digital audio data received by the server.

[1503] Specific actions taken: The server uses speech recognition technology (e.g., Google Speech-to-Text API) to convert the voice data into text data.

[1504] Server operation: The server sends the voice data to the speech recognition engine, which analyzes the voice and generates text data.

[1505] Output: The audio data is converted into "text data" and used in the next context analysis step.

[1506] Step 4: Contextual analysis of text data

[1507] Input: Text data returned from the speech recognition engine.

[1508] Specific actions taken: The server uses natural language processing techniques to analyze the context of the text data, including morphological analysis, semantic analysis, and sentiment analysis.

[1509] Server operation: The server analyzes the text data, performs calculations to identify the content and timing of the speaker's remarks, and extracts key keywords and sentiment.

[1510] Output: Information based on the contextual analysis is generated and used in the next step of backchannel generation.

[1511] Step 5: Selecting and generating backchannel phrases

[1512] Input: Information based on contextual analysis.

[1513] What happens: The server selects appropriate backchannels based on the contextual analysis results. The selected backchannels are either chosen from a pre-defined list or dynamically generated by a generative AI model.

[1514] Server operation: The server selects appropriate backchannel phrases based on the analysis results and automatically generates them in the form of chat messages.

[1515] Output: The selected backchannel phrase is generated as a chat message and used in the next sending step.

[1516] Step 6: Send a chat message

[1517] Input: An automatically generated chat backchannel message.

[1518] Specific actions taken: The server sends chat messages over the Internet to the user's device.

[1519] Server operation: The server initiates communication to send the generated chat message to the specified device. WebSocket or HTTP is used as the communication protocol.

[1520] Output: The chat message is sent to the device.

[1521] Step 7: View chat messages

[1522] Input: The chat message sent by the server.

[1523] Specific actions taken: Display chat messages received by the device on the screen.

[1524] Terminal Action: Displays a new message in a chat message window or similar, allowing all participants to see the message in real time.

[1525] Output: A backchannel message will be displayed in the chat window.

[1526] This series of processes allows users to receive appropriate responses from participants in real time during a lecture, making it easier for participants to become interested in the content of the lecture. This improves the quality of online communication and provides a more effective learning environment.

[1527] (Application example 1)

[1528] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1529] In online shopping virtual stores, customers often have difficulty getting a proper response in real time when they have questions or comments about products, which can lead to a poor customer experience and reduced purchasing intent.

[1530] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1531] In this invention, the server includes means for collecting the speaker's voice, means for converting the collected voice into text data, means for analyzing the context of the text data and selecting appropriate backchannels, means for sending the selected backchannels as chat messages, means for displaying the chat messages on a screen, means for collecting customer voice, converting it into text, analyzing the context and generating a response, and means for displaying the generated response message on a display device, thereby improving interaction with customers in real time and enabling a better shopping experience.

[1532] The "means for collecting the speaker's voice" refers to a device or method for collecting the speaker's voice in real time using a microphone device or the like and converting it into digital voice data.

[1533] The "means for converting collected voice data into text data" refers to a device or method for converting collected voice data into text information using voice recognition technology.

[1534] "Means for analyzing the context of text data and selecting appropriate responses" refers to a device or method that uses natural language processing technology to analyze the content and timing of statements contained in text data, and generates or selects appropriate responses or comments based on that context.

[1535] The "means for transmitting the selected backchannel as a chat message" refers to a device or method for transmitting the selected backchannel or response message in a chat format via a network.

[1536] The "means for displaying chat messages on a screen" refers to a device or method for displaying received chat messages on a user interface so that the user can check their contents.

[1537] "Means for collecting customer speech, converting it into text, analyzing the context, and generating a response" refers to a device or method that collects speech uttered by a customer, converts it into text information, analyzes the context, and automatically generates an appropriate response.

[1538] "Means for displaying the generated response message on a display device" refers to a device or method that outputs the generated response message to a display or other display device so that the user can visually confirm the message.

[1539] This invention is a system that automatically supports interactive communication in online lectures and virtual stores using speech recognition and context analysis. This system collects user speech, converts it into text in real time, analyzes the context, generates appropriate responses, and displays them on a display device. To function effectively, this system requires the following hardware and software:

[1540] Hardware and Software

[1541] Microphone devices: Used to collect audio data. This includes computers, smartphones, tablets, etc. with microphones.

[1542] Computer server: Receives the voice data and analyzes it using speech recognition and natural language processing technologies. It converts the voice data into text using Google Cloud's Speech-to-Text API and applies the GPT-3.5-turbo model using the transformers library.

[1543] Display device: A display device that shows chat messages and reply messages to users.

[1544] Data processing flow

[1545] 1. Audio collection: The device (user's smartphone or computer) collects the user's audio through a microphone device and transmits it to the server as digital audio data.

[1546] 2. Converting speech data to text: The server converts the speech data received into text using Google Cloud's Speech-to-Text API and prepares it for contextual analysis.

[1547] 3. Contextual analysis and response generation: The server uses natural language processing techniques to analyze the context of the text data, and uses the GPT-3.5-turbo model to generate appropriate responses based on that analysis.

[1548] 4. Creating and displaying a chat message or response message: The server sends the generated response to the terminal, which displays the message on its display device.

[1549] Specific examples

[1550] Example 1: Virtual store example

[1551] 1. Audio collection:

[1552] Customer: "Do you have this jacket in any other colors?"

[1553] Device: Collects audio from the microphone and sends it to the server.

[1554] 2. Text conversion:

[1555] Server: Sends the voice data to the voice recognition engine and receives the text data "Do you have this jacket in other colors?"

[1556] 3. Context analysis and response generation:

[1557] Server: Analyzes the text data using natural language processing technology and generates a response based on the context: "Of course we have it. Currently, it comes in three colors: black, blue, and red."

[1558] 4. Chat sending and display:

[1559] Server: Sends the generated response message to the terminal.

[1560] Terminal: Displays the response message on the user's screen.

[1561] Example prompt sentence:

[1562] Customer: Do you have this jacket in other colors?

[1563] context:

[1564] Generate a response.

[1565] The present invention improves customer experience in virtual stores and enables real-time interactive communication.

[1566] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1567] Step 1:

[1568] Audio collection: The device (user's smartphone or computer) collects the user's voice using a microphone device. When the user speaks, the microphone captures it and converts it into digital audio data in real time. This digital audio data is sent to the next processing step.

[1569] Input: User's voice

[1570] Output: Digital audio data

[1571] What it does: The microphone captures audio and the application converts it into a digital format.

[1572] Step 2:

[1573] Speech-to-text conversion: The device sends the collected digital speech data to a server, which converts the speech to text using Google Cloud's Speech-to-Text API and prepares the converted text for analysis.

[1574] Input: Digital audio data

[1575] Output: Text data

[1576] What happens: The server sends the audio data to the Google Cloud Speech-to-Text API, which converts the audio into text.

[1577] Step 3:

[1578] Contextual analysis and response generation: The server receives the text data and analyzes its context using natural language processing techniques. This analysis uses the transformers library and the GPT-3.5-turbo model. The generative AI model generates an appropriate response prompt from the analysis results.

[1579] Input: Text data

[1580] Output: Response message

[1581] Specific operation: The server sends a prompt to the GPT-3.5-turbo model using text data as input, and the model generates a response message.

[1582] Step 4:

[1583] Sending a response message: The server sends the generated response message to the terminal, which receives it and prepares it for display on the display device.

[1584] Input: Response message

[1585] Output: Response message sent to the terminal

[1586] Specific operation: The server sends a response message to the terminal over the network.

[1587] Step 5:

[1588] Display: The terminal displays the received response message on a display device, allowing the user to see the generated response in real time.

[1589] Input: Response message sent to the terminal

[1590] Output: Response message displayed on display device

[1591] Specific actions: The terminal renders the response message on the screen and displays it on the display device.

[1592] This enables real-time two-way communication between users and the system, improving the customer experience in the virtual store.

[1593] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1594] This invention is a system for promoting interactive communication between speakers and participants in online study groups and lectures. In particular, it combines an emotion engine to analyze the user's emotions and generate and display appropriate responses, enabling more lively communication.

[1595] This system is composed of the following major devices and technologies:

[1596] 1. Collecting speaker audio

[1597] The user (speaker) begins a lecture or explanation using a device with a microphone.

[1598] The terminal collects the speaker's voice in real time and transmits it to the server as digital voice data.

[1599] 2. Converting voice data to text

[1600] The server transmits the received voice data to a voice recognition engine, which converts the voice data into text data.

[1601] The speech recognition engine returns the converted text data to the server, which receives it.

[1602] 3. Context Analysis and Backchannel Generation

[1603] The server passes the text data to a natural language processing (NLP) module for contextual analysis.

[1604] The NLP module analyzes the context of the text data and determines appropriate interjections (e.g., "Uh-huh," "I see," "Great!", "Amazing!", etc.).

[1605] The server selects and generates appropriate responses based on the analysis results.

[1606] 4. Sentiment analysis and backchannel optimization

[1607] The server passes the collected voice data and text data to the emotion engine.

[1608] The emotion engine analyzes emotions from the user's voice and text, and selects appropriate responses based on the results.

[1609] For example, through emotion analysis, if a speaker is speaking enthusiastically, the system will generate many positive responses such as "Amazing!", but if the listeners are feeling stressed, it will generate neutral or empathetic responses such as "I see" or "Did you understand?"

[1610] 5. Sending and Viewing Chat Messages

[1611] The server sends the generated chat message to the device.

[1612] The terminal displays the sent chat message on the user's screen so that all participants can see the message.

[1613] Specific examples

[1614] Example 1: A speaker is explaining a new theory

[1615] 1. Audio collection

[1616] User (speaker): "Based on this theory, performance will improve significantly."

[1617] Device: Collects audio from the microphone and sends it to the server.

[1618] 2. Text Conversion

[1619] Server: Sends the speech data to the speech recognition engine and receives the text data, "Based on this theory, performance will be significantly improved."

[1620] 3. Context Analysis and Backchannel Generation

[1621] Server: Analyzes the text data using natural language processing technology and selects a "Like!" response based on the context.

[1622] Server: Generate a "Like" chat message.

[1623] 4. Sentiment analysis and backchannel optimization

[1624] Server: Passes voice and text data to the emotion engine.

[1625] Emotion Engine: If a speaker is speaking confidently, it analyzes positive emotions and selects additional back-channel responses like "Awesome!" to emphasize them.

[1626] Server: Also generates a "Awesome!" chat message.

[1627] 5. Chat sending and display

[1628] Server: Sends the generated "Like" and "Awesome" chat messages to the device.

[1629] Terminal: Chat messages are displayed on the user's screen.

[1630] This system will increase interactivity between speakers and participants in online study groups and lectures, and provide emotion-based feedback, resulting in more meaningful and effective communication.

[1631] The processing flow will be explained below.

[1632] Program processing steps

[1633] Step 1:

[1634] The user accesses the online study group link and enters their login information.

[1635] The terminal sends the entered login information to the server.

[1636] The server checks the authentication information against a database to verify that the participant is legitimate.

[1637] Step 2:

[1638] The user (speaker) begins speaking using a device with a microphone.

[1639] The device collects the speaker's voice in real time and transmits it to the server as digital audio data.

[1640] Step 3:

[1641] The server sends the received voice data to the voice recognition engine.

[1642] A voice recognition engine converts the voice data into text data.

[1643] The server receives the converted text data.

[1644] Step 4:

[1645] The server passes the text data to a natural language processing (NLP) module.

[1646] The NLP module analyzes the context of the text data and determines appropriate backchannel phrases.

[1647] The server selects the appropriate response based on the analysis results.

[1648] Step 5:

[1649] The server sends the collected voice data and converted text data to the emotion engine.

[1650] The emotion engine analyzes the user's emotions and selects the most appropriate response phrase based on the results.

[1651] The server generates the selected backchannel phrases and constructs them as chat messages.

[1652] Step 6:

[1653] The server sends the generated chat message to the device.

[1654] The terminal displays the sent chat message on the user's screen.

[1655] Specific examples

[1656] Example 1: A speaker is explaining a new theory

[1657] Step 1:

[1658] The user (speaker) says, "Based on this theory, performance will improve significantly."

[1659] The terminal collects the speaker's voice and sends it to the server.

[1660] Step 2:

[1661] The server receives the voice data and sends it to the voice recognition engine.

[1662] The speech recognition engine converts the speech into text and generates text data that says, "Based on this theory, performance will be significantly improved."

[1663] The server receives the conversion result.

[1664] Step 3:

[1665] The server passes the text data to the NLP module.

[1666] The NLP module analyzes the context and determines the "like" response.

[1667] The server selects "Likes" based on the analysis results.

[1668] Step 4:

[1669] The server sends the text data and voice data to the emotion engine.

[1670] The emotion engine analyzes the user's emotions and detects positive emotions.

[1671] The emotion engine also determines that the response "Amazing!" is appropriate and selects it.

[1672] The server generates "Like" and "Awesome" chat messages.

[1673] Step 5:

[1674] The server sends the generated chat message to the device.

[1675] The terminal displays the chat messages on the user's screen.

[1676] As shown in this example, the system converts the speaker's speech into text and generates and displays appropriate responses using natural language processing and sentiment analysis. This process improves interaction in online study groups and lectures, promoting two-way communication between speakers and participants.

[1677] Example 2

[1678] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1679] Online study groups and lectures face a lack of interactive communication between speakers and participants. In particular, there is a lack of a way to automatically generate appropriate responses and provide emotion-based feedback to the speaker. This makes it difficult for participants to respond appropriately to the content of the lecture, and the lecture tends to become a one-way communication of information.

[1680] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1681] In this invention, the server includes means for collecting the speaker's voice, means for converting the collected voice into text data, means for analyzing the context of the text data and selecting appropriate backchannels, means for optimizing the selected backchannels based on emotion analysis, means for sending the optimized backchannels as chat messages, and means for displaying the chat messages on a screen. This enables interactive communication with participants in online study groups and lectures by automatically generating appropriate backchannels based on emotion.

[1682] "Audio collection means" refers to a device or technology that captures the speaker's voice as digital data and transmits that data to a server.

[1683] The "means for converting voice into text data" refers to a voice recognition technology or system that automatically converts collected voice data into text data.

[1684] "Means for analyzing context and selecting appropriate backchannels" refers to natural language processing (NLP) technology or systems that analyze the content of text data and determine appropriate backchannel phrases.

[1685] The "means for optimizing based on emotion analysis" is an emotion engine or algorithm that analyzes the user's emotions and selects and optimizes appropriate responses based on the results.

[1686] The "means for transmitting backchannels as chat messages" refers to a communication technology or system that transmits the generated backchannels to a terminal in the form of a chat message.

[1687] The "means for displaying chat messages on a screen" refers to a technology or system for displaying sent chat messages on the terminals of participants.

[1688] This invention is a system for promoting interactive communication between speakers and participants in online study groups and lectures. In particular, by combining an emotion engine to analyze the user's emotions and generate and display appropriate responses, more lively communication is possible.

[1689] Hardware and software used

[1690] 1. A device with a microphone (e.g., a headset or smartphone)

[1691] 2. Server (including speech recognition engine, natural language processing engine, emotion engine)

[1692] 3. Devices (PC, smartphone, tablet)

[1693] Detailed explanation of the process

[1694] First, the user (speaker) begins their lecture using a device with a microphone. The device collects the speaker's voice in real time and transmits it to the server as digital audio data. For example, the speaker's utterance, "Good morning, today I will talk about the future of AI," is collected by the microphone and transmitted as digital data to the server.

[1695] Next, the server sends the received voice data to a speech recognition engine and converts the voice data into text data. Specifically, it uses the Google Cloud Speech-to-Text API to convert the voice data into text data such as "Good morning, today I'll talk about the future of AI." The server receives this converted text data.

[1696] The server then passes this text data to a natural language processing (NLP) module for context analysis. Using Hugging Face's Transformers library, the server analyzes the context of the text data and determines appropriate backchannel phrases. For example, it selects "Interesting!" from the context "Today we'll talk about the future of AI."

[1697] The server then passes the collected voice and text data to the emotion engine. Using IBM Watson's Tone Analyzer, the engine analyzes the user's emotions and selects the most appropriate backchannel based on the results. For example, if the speaker is analyzed as speaking with confidence, the engine selects an additional positive backchannel such as "That's great! I'd like to hear more!" By sending this backchannel, it becomes possible to provide feedback based on the speaker's emotions.

[1698] In the final stage, the server sends the generated chat message to the device, which displays it on the user's screen so that all participants can see it. For example, chat messages such as "Interesting!" and "Awesome, I'd love to hear more!" are displayed and shared with participants in real time.

[1699] Specific examples

[1700] Example: A speaker is explaining a new theory.

[1701] 1. User (speaker): "Based on this theory, performance will improve significantly."

[1702] 2. Device: Collects audio from the microphone and sends it to the server.

[1703] 3. Server: Sends the audio data to the Google Cloud Speech-to-Text API and receives it as text data.

[1704] 4. Server: The text data is analyzed using Hugging Face's Transformers library to determine whether it is a "Like!"

[1705] 5. Server: Use IBM Watson's Tone Analyzer to analyze emotions and select additional "Amazing!" responses.

[1706] 6. Server: Sends "Like" and "Awesome" chat messages to the device.

[1707] 7. Terminal: Chat messages are displayed on the user's screen so that participants can see them.

[1708] Prompt Sentence Examples

[1709] The speaker will talk about "The Future of AI Technology." Part of the lecture will include the following:

[1710] Speaker remarks: "These technologies have the potential to profoundly change the way we live our lives."

[1711] System-based text conversion: "These technologies have the potential to significantly change our lives."

[1712] System analysis: "Great!"

[1713] Sentiment analysis by the system: "Amazing!"

[1714] Expected output:

[1715] When the speaker says, "These technologies have the potential to dramatically change our lives," the responses "Amazing!" and "Amazing!" are displayed.

[1716] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1717] Step 1:

[1718] Collecting speaker audio

[1719] User (speaker): Starts speaking using a device with a microphone.

[1720] Input: Speaker audio from a device with a microphone.

[1721] Output: Audio data in digital format.

[1722] Specific behavior:

[1723] The speaker says, "Good morning, today I'll be talking about the future of AI." The device (computer or smartphone app) captures the voice with its built-in microphone, converts the voice data into digital format, and transmits it to a server in real time.

[1724] Step 2:

[1725] Converting audio data to text

[1726] Server: Sends the received voice data to the speech recognition engine.

[1727] Input: Audio data in digital format.

[1728] Output: Text data.

[1729] Specific behavior:

[1730] The server sends the received voice data to the Google Cloud Speech-to-Text API and converts it into text data such as "Good morning, today I'll talk about the future of AI." The converted text data is then received by the server.

[1731] Step 3:

[1732] Context analysis and backchannel generation

[1733] Server: Passes the text data to a natural language processing (NLP) module for contextual analysis.

[1734] Input: The converted text data.

[1735] Output: An appropriate backchannel phrase.

[1736] Specific behavior:

[1737] The server passes the text data to an NLP module using Hugging Face's Transformers library. The NLP module analyzes the context, "Today we'll talk about the future of AI," and selects an appropriate back-channel phrase, such as "Interesting!" The server then generates a chat message saying, "Interesting!"

[1738] Step 4:

[1739] Sentiment analysis and backchannel optimization

[1740] Server: Passes the collected voice and text data to the emotion engine.

[1741] Input: Audio and text data.

[1742] Output: Optimized backchannel phrases.

[1743] Specific behavior:

[1744] The server passes the audio and text data to IBM Watson's Tone Analyzer. If the emotion engine determines that the speaker is speaking confidently, it selects a positive response such as, "Great, I'd love to hear more!" The server then generates this new response as a chat message.

[1745] Step 5:

[1746] Sending and viewing chat messages

[1747] Server: Sends generated chat messages to devices.

[1748] Input: Optimized backchannel phrase.

[1749] Output: The chat messages that are displayed on the user's screen.

[1750] Specific behavior:

[1751] The server sends the generated chat messages, such as "Interesting!" and "Amazing, I'd love to hear more!", to the device. The device (participant's PC or smartphone) displays the chat messages on the user's screen. All participants can view the messages in real time through the chat window.

[1752] (Application example 2)

[1753] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1754] In online study groups and lectures, there is a lack of interactive communication between speakers and participants, so more active and meaningful communication is required. In addition, there is a challenge in providing appropriate feedback on audience emotions and reactions to improve the quality of real-time interactions.

[1755] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting the speaker's voice, means for converting the collected voice into text data, means for analyzing the context of the text data and selecting appropriate backchannels, means for sending the selected backchannels as chat messages, means for displaying the chat messages on a screen, means for collecting viewer comments and analyzing the context and emotions of the comments, and means for generating and displaying appropriate backchannels and reactions to the viewer based on the analysis results. This allows for real-time exchange of emotion-based feedback between the speaker, participants, viewers, etc., enabling more fulfilling interactive communication.

[1756] A "speaker" is a person who speaks at an online study group or lecture.

[1757] "Means for collecting audio" refers to devices and methods for recording the speaker's speech and transmitting it as digital data to a server.

[1758] "Means for converting voice into text data" refers to the technology or method for analyzing collected voice data and converting it into text format.

[1759] "Means for analyzing the context of text data" refers to natural language processing techniques and methods for analyzing text data and understanding its content and meaning.

[1760] "Means for selecting backchannels" refers to techniques and methods for selecting appropriate backchannels and responses based on the results of context analysis and emotion analysis.

[1761] "Means for sending as a chat message" refers to a technique or method for sending the generated backchannels or responses as an electronic message.

[1762] The "means for displaying chat messages on a screen" refers to a technique or method for displaying the sent messages on a display device.

[1763] "Means for collecting viewer comments" refers to techniques and methods for collecting comments from viewers in real time.

[1764] "Means for analyzing the context and sentiment of comments" refers to natural language processing and sentiment analysis technologies that analyze comments from viewers and understand their content and emotional state.

[1765] "Means for generating appropriate responses and reactions to viewers based on the analysis results" refers to technology or methods that automatically generate appropriate responses and reactions to viewers' comments based on the analysis results.

[1766] This invention provides a system for promoting interactive communication between a lecturer and an audience in an online study group or lecture meeting. The operation of this system will be described in detail below.

[1767] 1. Audio collection and conversion

[1768] A device with a microphone is used to collect the speaker's voice. The user (speaker) starts their lecture or explanation, and the device collects the speaker's voice in real time and sends this audio data in digital format to a server. The server then converts the received audio data into text data using a speech recognition engine (e.g., Google Speech-to-Text API).

[1769] 2. Context Analysis and Backchannel Generation

[1770] The server passes the text data to a natural language processing (NLP) module (such as SpaCy or NLTK) to analyze the context. The NLP module analyzes the context of the text data and determines appropriate backchannel phrases (e.g., "Uh-huh," "I see," "Great!", "Amazing!"). Based on the analysis results, the server generates appropriate backchannels.

[1771] 3. Collection and analysis of viewer comments

[1772] The server uses real-time communication technology (e.g., WebSocket) to collect comments from viewers in real time. After collecting the comments, these comments are also passed to the NLP module to analyze the context and sentiment. For sentiment analysis, an emotion engine (e.g., IBM Watson Tone Analyzer) is used.

[1773] 4. Reaction generation and display

[1774] The server generates appropriate responses and reactions for the audience based on the analysis results. These responses and reactions are sent as chat messages, which are then displayed on the screens of the user (speaker) and audience members. This allows the speaker to see audience reactions in real time, and audience members can receive feedback on their own comments.

[1775] Specific examples

[1776] Below is a specific example scenario of a live stream where a speaker is talking about protecting the environment.

[1777] 1. Speaker: "We are being asked to reduce our use of plastic by 50% over the next 10 years."

[1778] 2. Real-time analysis and backchannel generation:

[1779] The speech recognition engine converts speech into text.

[1780] The NLP module analyzes the context and generates an appropriate response: "That's important!"

[1781] The sentiment analysis engine analyzes and emphasizes the speaker's passion, adding "What an amazing initiative!"

[1782] 3. Feedback display:

[1783] The speaker sees these responses in real time and continues speaking with even more enthusiasm.

[1784] Prompt Sentence Examples

[1785] Below is an example of a prompt sentence that is input to a generative AI model.

[1786] Speaker's audio data

[1787] There is a call to reduce plastic use by 50% over the next 10 years.

[1788] prompt

[1789] Generate appropriate responses to what this speaker has said based on audience sentiment analysis.

[1790] Generated responses (example)

[1791] That's important!

[1792] What an amazing initiative!

[1793] This allows for real-time emotional feedback between the speaker and the audience, enabling richer, more interactive communication.

[1794] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1795] Step 1:

[1796] The speaker (user) starts a lecture or explanation using a device with a microphone. The device collects the speaker's voice in real time, converts this voice data into a digital format, and sends it to the server.

[1797] Input: Speaker's audio data

[1798] Output: Digital audio data

[1799] How it works: The device collects audio through a microphone, converts it into digital form, and sends it over the network to a server.

[1800] Step 2:

[1801] The server passes the received voice data to a voice recognition engine (for example, Google Speech-to-Text API) and converts the voice data into text data.

[1802] Input: Digital audio data

[1803] Output: Text data

[1804] Specific operation: The server sends audio data to the Google Speech-to-Text API and receives the converted text data.

[1805] Step 3:

[1806] The server passes the text data to a natural language processing (NLP) module (e.g., SpaCy or NLTK) for context analysis, which then determines the appropriate backchannel phrases.

[1807] Input: Text data

[1808] Output: Context analysis results and backchannel phrases

[1809] What it does: The server passes the text data to the NLP module, which generates appropriate back-channel responses (e.g., "Uh-huh," "I see," "Great!").

[1810] Step 4:

[1811] The server formats the generated backchannel phrases as chat messages and sends them to the terminal.

[1812] Input: Backchannel phrase

[1813] Output: Chat message

[1814] Specific operation: The server converts the generated backchannel phrases into chat messages and sends them to each viewer's device.

[1815] Step 5:

[1816] The terminal displays the received chat messages on the screen so that the speaker and the audience can see the messages.

[1817] Input: Chat message

[1818] Output: Chat messages displayed

[1819] Specific operation: The device displays the received chat message on the user's screen.

[1820] Step 6:

[1821] The server collects comments from viewers in real time and passes these comments to a natural language processing (NLP) module for context and sentiment analysis.

[1822] Input: Viewer comments

[1823] Output: Contextual and sentiment analysis results

[1824] Specific operation: The server collects comments using real-time communication technology (such as WebSocket) and passes them to the NLP module and emotion engine for analysis.

[1825] Step 7:

[1826] The server generates appropriate responses and reactions based on the viewers' comments, which are then formatted as chat messages and sent to the device.

[1827] Input: Contextual and sentiment analysis results

[1828] Output: Chat message

[1829] Specific operation: The server generates responses and reactions to the comments based on the analysis results, formats them as chat messages, and sends them.

[1830] Step 8:

[1831] The device displays the generated responses and reactions on the viewer's screen, allowing them to see the feedback in real time.

[1832] Input: Chat message

[1833] Output: The displayed responses and reactions

[1834] What it does: The device displays the received responses and reactions on the viewer's screen, providing real-time feedback.

[1835] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1836] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1837] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1838] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1839] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1840] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1841] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1842] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1843] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1844] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1845] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1846] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1847] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1848] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1849] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1850] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1851] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1852] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1853] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1854] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1855] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1856] The following is further disclosed regarding the above embodiment.

[1857] (Claim 1)

[1858] a means for collecting audio of the speaker;

[1859] A means for converting the collected voice into text data;

[1860] A means for analyzing the context of text data and selecting appropriate responses;

[1861] a means for sending the selected backchannel as a chat message;

[1862] a means for displaying chat messages on the screen;

[1863] A system including:

[1864] (Claim 2)

[1865] 10. The system of claim 1, further comprising means for converting speech to text data using speech recognition technology.

[1866] (Claim 3)

[1867] 10. The system of claim 1, further comprising means for analyzing the context of the text data using natural language processing techniques.

[1868] "Example 1"

[1869] (Claim 1)

[1870] a means for collecting audio of the speaker;

[1871] A means for converting the collected voice into text data;

[1872] A means for analyzing the context of text data and selecting appropriate responses;

[1873] a means for sending the selected backchannel as a chat message;

[1874] a means for displaying chat messages on the screen;

[1875] A system including:

[1876] (Claim 2)

[1877] 10. The system of claim 1, further comprising means for converting speech to text data using speech recognition technology.

[1878] (Claim 3)

[1879] 10. The system of claim 1, further comprising means for analyzing the context of the text data using natural language processing techniques.

[1880] "Application Example 1"

[1881] (Claim 1)

[1882] a means for collecting audio of the speaker;

[1883] A means for converting the collected voice into text data;

[1884] A means for analyzing the context of text data and selecting appropriate responses;

[1885] a means for sending the selected backchannel as a chat message;

[1886] a means for displaying chat messages on the screen;

[1887] A means for collecting customer speech, converting it into text, and analyzing the context to generate a response;

[1888] means for displaying the generated response message on a display device;

[1889] A system including:

[1890] (Claim 2)

[1891] 10. The system of claim 1, further comprising means for converting speech to text data using speech recognition technology.

[1892] (Claim 3)

[1893] 10. The system of claim 1, further comprising means for analyzing the context of the text data using natural language processing techniques.

[1894] "Example 2: Combining Emotion Engines"

[1895] (Claim 1)

[1896] a means for collecting audio of the speaker;

[1897] A means for converting the collected voice into text data;

[1898] A means for analyzing the context of text data and selecting appropriate responses;

[1899] A means for optimizing the selected backchannels based on emotion analysis;

[1900] A means of sending optimized backchannels as chat messages;

[1901] a means for displaying chat messages on the screen;

[1902] A system including:

[1903] (Claim 2)

[1904] 10. The system of claim 1, further comprising means for converting speech to text data using speech recognition technology.

[1905] (Claim 3)

[1906] 10. The system of claim 1, further comprising means for analyzing the context of the text data using natural language processing techniques.

[1907] "Application example 2 when combining emotion engines"

[1908] (Claim 1)

[1909] a means for collecting audio of the speaker;

[1910] A means for converting the collected voice into text data;

[1911] A means for analyzing the context of text data and selecting appropriate responses;

[1912] a means for sending the selected backchannel as a chat message;

[1913] a means for displaying chat messages on the screen;

[1914] a means of collecting viewer comments and analyzing the context and sentiment of those comments;

[1915] A means for generating and displaying appropriate responses and reactions to viewers based on the analysis results;

[1916] A system including:

[1917] (Claim 2)

[1918] 10. The system of claim 1, further comprising means for converting speech to text data using speech recognition technology.

[1919] (Claim 3)

[1920] 10. The system of claim 1, further comprising means for analyzing the context of the text data using natural language processing techniques. [Explanation of symbols]

[1921] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. a means for collecting audio of the speaker; A means for converting the collected voice into text data; A means for analyzing the context of text data and selecting appropriate responses; a means for sending the selected backchannel as a chat message; a means for displaying chat messages on the screen; A system including:

2. 10. The system of claim 1, further comprising means for converting speech to text data using speech recognition technology.

3. The system of claim 1 , further comprising means for analyzing the context of the text data using natural language processing techniques.

Images